agora inbox for [email protected]
help / color / mirror / Atom feedRe: Duplicate Workers entries in some EXPLAIN plans
288+ messages / 5 participants
[nested] [flat]
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-14 14:44 Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-14 14:44 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
This is a high level review only. However, seeing that there is a conflict and does not merge cleanly after commit <b925a00f4ef>, I return to Author.
To be fair the resolution seems quite straight forward and I took the liberty of applying the necessary changes to include the new element of ExplainState introduced in the above commit, namely hide_workers. However since the author might have a different idea on how to incorporate this change I leave it up to him.
Another very high level comment is the introduction of a new test file, namely explain. Seeing `explain.sql` in the tests suits, personally and very opinion based, I would have expected the whole spectrum of the explain properties to be tested. However only a slight fraction of the functionality is tested. Since this is a bit more of a personal opinion, I don't expect any changes unless the author happens to agree.
Other than these minor nitpick, the code seems clear, concise and does what it says on the box. It follows the comments in the discussion thread(s) and solves a real issue.
Please have a look on how commit <b925a00f4ef> affects this patch and rebase.
The new status of this patch is: Waiting on Author
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-15 07:22 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-15 07:22 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
Thanks for the review! I looked at <b925a00f4ef> and rebased the patch
on current master, ac5bdf6.
I introduced a new test file because this bug is specifically about
EXPLAIN output (as opposed to query execution or planning
functionality), and it didn't seem like a test would fit in any of the
other files. I focused on testing just the behavior around this
specific bug (and fix). I think eventually we should probably test
other more fundamental EXPLAIN features (and I'm happy to contribute
to that) in that file, but that seems outside of the scope of this
patch.
Any thoughts on what we should do with text mode output (which is
untouched right now)? The output Andres proposed above makes sense to
me, but I'd like to get more input.
Attachments:
[text/x-patch] merge-explain-worker-output-v2.patch (18.9K, ../../CAOtHd0CC1iVG0ZA08onswgiOU9++Qhec0WRtECk-7zWM+nE2Rg@mail.gmail.com/2-merge-explain-worker-output-v2.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..96a8973884 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -100,7 +100,7 @@ static void show_sortorder_options(StringInfo buf, Node *sortexpr,
Oid sortOperator, Oid collation, bool nullsFirst);
static void show_tablesample(TableSampleClause *tsc, PlanState *planstate,
List *ancestors, ExplainState *es);
-static void show_sort_info(SortState *sortstate, ExplainState *es);
+static void show_sort_info(SortState *sortstate, StringInfo* worker_strs, ExplainState *es);
static void show_hash_info(HashState *hashstate, ExplainState *es);
static void show_tidbitmap_info(BitmapHeapScanState *planstate,
ExplainState *es);
@@ -289,6 +289,7 @@ NewExplainState(void)
es->costs = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
+ es->root_str = es->str;
return es;
}
@@ -1090,9 +1091,20 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ StringInfo *worker_strs = NULL;
int save_indent = es->indent;
bool haschildren;
+ /* Prepare per-worker output */
+ if (es->analyze && planstate->worker_instrument) {
+ int num_workers = planstate->worker_instrument->num_workers;
+ int n;
+ worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
+ for (n = 0; n < num_workers; n++) {
+ worker_strs[n] = makeStringInfo();
+ }
+ }
+
switch (nodeTag(plan))
{
case T_Result:
@@ -1357,6 +1369,64 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyBool("Parallel Aware", plan->parallel_aware, es);
}
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && planstate->worker_instrument)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+ int n;
+
+ for (n = 0; n < w->num_workers; ++n)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ appendStringInfo(es->str, "Worker %d: ", n);
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ es->indent++;
+ if (es->buffers)
+ show_buffer_usage(es, &instrument->bufusage);
+ es->indent--;
+ }
+ else
+ {
+ ExplainOpenWorker(worker_strs[n], es);
+
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+ }
+
+ if (es->format != EXPLAIN_FORMAT_TEXT) {
+ ExplainCloseWorker(es);
+ }
+ }
+
switch (nodeTag(plan))
{
case T_SeqScan:
@@ -1856,7 +1926,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
break;
case T_Sort:
show_sort_keys(castNode(SortState, planstate), ancestors, es);
- show_sort_info(castNode(SortState, planstate), es);
+ show_sort_info(castNode(SortState, planstate), worker_strs, es);
break;
case T_MergeAppend:
show_merge_append_keys(castNode(MergeAppendState, planstate),
@@ -1885,74 +1955,30 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare worker buffer usage */
+ if (es->buffers && es->analyze && es->verbose && !es->hide_workers
+ && planstate->worker_instrument && es->format != EXPLAIN_FORMAT_TEXT)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
int n;
for (n = 0; n < w->num_workers; ++n)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
-
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
-
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(worker_strs[n], es);
+ show_buffer_usage(es, &instrument->bufusage);
}
+ ExplainCloseWorker(es);
+ }
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ /* Show worker detail */
+ if (planstate->worker_instrument) {
+ ExplainFlushWorkers(worker_strs, planstate->worker_instrument->num_workers, es);
}
/* Get ready to display the child plans */
@@ -2552,7 +2578,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
* If it's EXPLAIN ANALYZE, show tuplesort stats for a sort node
*/
static void
-show_sort_info(SortState *sortstate, ExplainState *es)
+show_sort_info(SortState *sortstate, StringInfo *worker_strs, ExplainState *es)
{
if (!es->analyze)
return;
@@ -2593,7 +2619,6 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2620,21 +2645,15 @@ show_sort_info(SortState *sortstate, ExplainState *es)
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
+ ExplainOpenWorker(worker_strs[n], es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ if (es->format != EXPLAIN_FORMAT_TEXT) {
+ ExplainCloseWorker(es);
+ }
}
}
@@ -3751,6 +3770,62 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
}
+/*
+ * Begin output for a specific worker. If we are currently producing output
+ * for another worker, close that worker automatically.
+ */
+void
+ExplainOpenWorker(StringInfo worker_str, ExplainState *es)
+{
+ /*
+ * We indent twice--once for the "Worker" group and once for the "Workers"
+ * group--but only need to adjust indentation once for each string of
+ * ExplainOpenWorker calls.
+ */
+ if (es->str == es->root_str) {
+ es->indent += 2;
+ }
+
+ es->print_workers = true;
+ es->str = worker_str;
+}
+
+/*
+ * Explicitly end output for the current worker and resume output
+ * for the root of the node.
+ */
+void
+ExplainCloseWorker(ExplainState *es)
+{
+ es->indent -= 2;
+ es->str = es->root_str;
+}
+
+/*
+ * Flush output registered so far to the output buffer and deallocate
+ * mechanism for writing to workers.
+ */
+void
+ExplainFlushWorkers(StringInfo* worker_strs, int num_workers, ExplainState *es)
+{
+ int i;
+
+ if (!es->print_workers) {
+ return;
+ }
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (i = 0; i < num_workers; i++) {
+ ExplainOpenGroup("Worker", NULL, true, es);
+ ExplainPropertyInteger("Worker Number", NULL, i, es);
+ appendStringInfoString(es->str, worker_strs[i]->data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+ // do we have any other cleanup to do?
+ es->print_workers = false;
+}
+
/*
* Emit a "dummy" group that never has any members.
*
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7c47..2919b2780d 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -27,7 +27,7 @@ typedef enum ExplainFormat
typedef struct ExplainState
{
- StringInfo str; /* output buffer */
+ StringInfo str; /* current output buffer */
/* options */
bool verbose; /* be verbose */
bool analyze; /* print actual times */
@@ -40,6 +40,8 @@ typedef struct ExplainState
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
List *grouping_stack; /* format-specific grouping state */
+ bool print_workers; /* whether current node has worker metadata */
+ StringInfo root_str; /* root output buffer */
/* state related to the current plan tree (filled by ExplainPrintPlan) */
PlannedStmt *pstmt; /* top of plan */
List *rtable; /* range table */
@@ -111,4 +113,9 @@ extern void ExplainOpenGroup(const char *objtype, const char *labelname,
extern void ExplainCloseGroup(const char *objtype, const char *labelname,
bool labeled, ExplainState *es);
+extern void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
+extern void ExplainCloseWorker(ExplainState *es);
+extern void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
+
+
#endif /* EXPLAIN_H */
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
new file mode 100644
index 0000000000..db9914db7b
--- /dev/null
+++ b/src/test/regress/expected/explain.out
@@ -0,0 +1,75 @@
+--
+-- EXPLAIN
+--
+-- helper functions for examining json explain output
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+ explain_workers
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ { "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Sort Method" : "quicksort", "Sort Space Used" : 0, "Sort Space Type" : "Memory", "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }], "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }] }] }] }
+(1 row)
+
+reset force_parallel_mode;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index d33a4e143d..d2b17dd3ea 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,7 +112,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
# event triggers cannot run concurrently with any test that runs DDL
test: event_trigger
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index f86f5c5682..acba391332 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -193,6 +193,7 @@ test: indexing
test: partition_aggregate
test: partition_info
test: tuplesort
+test: explain
test: event_trigger
test: fast_default
test: stats
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
new file mode 100644
index 0000000000..5fb6e7d2e1
--- /dev/null
+++ b/src/test/regress/sql/explain.sql
@@ -0,0 +1,77 @@
+--
+-- EXPLAIN
+--
+
+-- helper functions for examining json explain output
+
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+
+reset force_parallel_mode;
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-15 10:11 Georgios Kokolatos <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-15 10:11 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: tested, failed
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
The current version of the patch (v2) applies cleanly and does what it says on the box.
> Any thoughts on what we should do with text mode output (which is
untouched right now)? The output Andres proposed above makes sense to
me, but I'd like to get more input.
Nothing to add beyond what is stated in the thread.
> Sort Method: external merge Disk: 4920kB
> Buffers: shared hit=682 read=10188, temp read=1415 written=2101
> Worker 0: actual time=130.058..130.324 rows=1324 loops=1
> Sort Method: external merge Disk: 5880kB
> Buffers: shared hit=337 read=3489, temp read=505 written=739
> Worker 1: actual time=130.273..130.512 rows=1297 loops=1
> Buffers: shared hit=345 read=3507, temp read=505 written=744
> Sort Method: external merge Disk: 5920kB
This proposal seems like a fitting approach. Awaiting for v3 which
will include the text version. May I suggest a format yaml test case?
Just to make certain that no regressions occur in the future.
Thanks,
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-15 17:12 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-15 17:12 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
Sounds good, I'll try that format. Any idea how to test YAML? With the
JSON format, I was able to rely on Postgres' own JSON-manipulating
functions to strip or canonicalize fields that can vary across
executions--I can't really do that with YAML. Or should I run EXPLAIN
with COSTS OFF, TIMING OFF, SUMMARY OFF and assume that for simple
queries the BUFFERS output (and other fields I can't turn off like
Sort Space Used) *is* going to be stable?
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-16 14:07 Georgios Kokolatos <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-16 14:07 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
> Sounds good, I'll try that format. Any idea how to test YAML? With the
> JSON format, I was able to rely on Postgres' own JSON-manipulating
> functions to strip or canonicalize fields that can vary across
> executions--I can't really do that with YAML.
Yes, this approach was clear in the patch and works great with Json. Also
you are correct, this can not be done with YAML. I spend a bit of time to
look around and I could not find any tests really on yaml format.
> Or should I run EXPLAIN
> with COSTS OFF, TIMING OFF, SUMMARY OFF and assume that for simple
> queries the BUFFERS output (and other fields I can't turn off like
> Sort Space Used) *is* going to be stable?
I have to admit with the current diff tool used in pg_regress, this is not possible.
I am pretty certain that it *is not* going to be stable. Not for long anyways.
I withdraw my suggestion for YAML and currently awaiting for TEXT format only.
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-21 08:48 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-21 08:48 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
TEXT format was tricky due to its inconsistencies, but I think I have
something working reasonably well. I added a simple test for TEXT
format output as well, using a similar approach as the JSON format
test, and liberally regexp_replacing away any volatile output. I
suppose in theory we could do this for YAML, too, but I think it's
gross enough not to be worth it, especially given the high similarity
of all the structured outputs.
Attachments:
[text/x-patch] merge-explain-worker-output-v3.patch (23.3K, ../../CAOtHd0AmrNM4nfy19MZJZhxPSfa5OQPwOg96HivKvnO_CiajWg@mail.gmail.com/2-merge-explain-worker-output-v3.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..c3c27e13a1 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -100,7 +100,7 @@ static void show_sortorder_options(StringInfo buf, Node *sortexpr,
Oid sortOperator, Oid collation, bool nullsFirst);
static void show_tablesample(TableSampleClause *tsc, PlanState *planstate,
List *ancestors, ExplainState *es);
-static void show_sort_info(SortState *sortstate, ExplainState *es);
+static void show_sort_info(SortState *sortstate, StringInfo* worker_strs, ExplainState *es);
static void show_hash_info(HashState *hashstate, ExplainState *es);
static void show_tidbitmap_info(BitmapHeapScanState *planstate,
ExplainState *es);
@@ -289,6 +289,7 @@ NewExplainState(void)
es->costs = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
+ es->root_str = es->str;
return es;
}
@@ -1090,9 +1091,20 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ StringInfo *worker_strs = NULL;
int save_indent = es->indent;
bool haschildren;
+ /* Prepare per-worker output */
+ if (es->analyze && planstate->worker_instrument) {
+ int num_workers = planstate->worker_instrument->num_workers;
+ int n;
+ worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
+ for (n = 0; n < num_workers; n++) {
+ worker_strs[n] = makeStringInfo();
+ }
+ }
+
switch (nodeTag(plan))
{
case T_Result:
@@ -1357,6 +1369,58 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyBool("Parallel Aware", plan->parallel_aware, es);
}
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && planstate->worker_instrument)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+ int n;
+
+ for (n = 0; n < w->num_workers; ++n)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ ExplainOpenWorker(worker_strs[n], es);
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ if (es->buffers)
+ show_buffer_usage(es, &instrument->bufusage);
+ }
+ else
+ {
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+ }
+
+ ExplainCloseWorker(es);
+ }
+
switch (nodeTag(plan))
{
case T_SeqScan:
@@ -1856,7 +1920,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
break;
case T_Sort:
show_sort_keys(castNode(SortState, planstate), ancestors, es);
- show_sort_info(castNode(SortState, planstate), es);
+ show_sort_info(castNode(SortState, planstate), worker_strs, es);
break;
case T_MergeAppend:
show_merge_append_keys(castNode(MergeAppendState, planstate),
@@ -1885,74 +1949,30 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare worker buffer usage */
+ if (es->buffers && es->analyze && es->verbose && !es->hide_workers
+ && planstate->worker_instrument && es->format != EXPLAIN_FORMAT_TEXT)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
int n;
for (n = 0; n < w->num_workers; ++n)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
-
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
-
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(worker_strs[n], es);
+ show_buffer_usage(es, &instrument->bufusage);
}
+ ExplainCloseWorker(es);
+ }
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ /* Show worker detail */
+ if (planstate->worker_instrument) {
+ ExplainFlushWorkers(worker_strs, planstate->worker_instrument->num_workers, es);
}
/* Get ready to display the child plans */
@@ -2552,7 +2572,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
* If it's EXPLAIN ANALYZE, show tuplesort stats for a sort node
*/
static void
-show_sort_info(SortState *sortstate, ExplainState *es)
+show_sort_info(SortState *sortstate, StringInfo *worker_strs, ExplainState *es)
{
if (!es->analyze)
return;
@@ -2593,7 +2613,7 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
+ bool indent = es->print_workers;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2609,32 +2629,26 @@ show_sort_info(SortState *sortstate, ExplainState *es)
spaceType = tuplesort_space_type_name(sinstrument->spaceType);
spaceUsed = sinstrument->spaceUsed;
+ ExplainOpenWorker(worker_strs[n], es);
+
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ if (indent)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ }
appendStringInfo(es->str,
"Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ ExplainCloseWorker(es);
}
}
@@ -3751,6 +3765,75 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
}
+/*
+ * Begin output for a specific worker. If we are currently producing output
+ * for another worker, close that worker automatically.
+ */
+void
+ExplainOpenWorker(StringInfo worker_str, ExplainState *es)
+{
+ /*
+ * For structured formats, we indent twice--once for the "Worker" group
+ * and once for the "Workers" group--but only need to adjust indentation
+ * once for each string of ExplainOpenWorker calls. For text format, we
+ * just indent once, to add worker info on the next worker line.
+ */
+ if (es->str == es->root_str)
+ {
+ es->indent += es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+ }
+
+ es->print_workers = true;
+ es->str = worker_str;
+}
+
+/*
+ * Explicitly end output for the current worker and resume output
+ * for the root of the node.
+ */
+void
+ExplainCloseWorker(ExplainState *es)
+{
+ /* as above, the text format has different indentation */
+ es->indent -= es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+
+ es->str = es->root_str;
+}
+
+/*
+ * Flush output registered so far to the output buffer and deallocate
+ * mechanism for writing to workers.
+ */
+void
+ExplainFlushWorkers(StringInfo* worker_strs, int num_workers, ExplainState *es)
+{
+ int i;
+
+ if (!es->print_workers) {
+ return;
+ }
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (i = 0; i < num_workers; i++) {
+ ExplainOpenGroup("Worker", NULL, true, es);
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ appendStringInfo(es->str, "Worker %d: ", i);
+ }
+ else
+ {
+ ExplainPropertyInteger("Worker Number", NULL, i, es);
+ }
+
+ appendStringInfoString(es->str, worker_strs[i]->data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+ // do we have any other cleanup to do?
+ es->print_workers = false;
+}
+
/*
* Emit a "dummy" group that never has any members.
*
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7c47..2919b2780d 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -27,7 +27,7 @@ typedef enum ExplainFormat
typedef struct ExplainState
{
- StringInfo str; /* output buffer */
+ StringInfo str; /* current output buffer */
/* options */
bool verbose; /* be verbose */
bool analyze; /* print actual times */
@@ -40,6 +40,8 @@ typedef struct ExplainState
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
List *grouping_stack; /* format-specific grouping state */
+ bool print_workers; /* whether current node has worker metadata */
+ StringInfo root_str; /* root output buffer */
/* state related to the current plan tree (filled by ExplainPrintPlan) */
PlannedStmt *pstmt; /* top of plan */
List *rtable; /* range table */
@@ -111,4 +113,9 @@ extern void ExplainOpenGroup(const char *objtype, const char *labelname,
extern void ExplainCloseGroup(const char *objtype, const char *labelname,
bool labeled, ExplainState *es);
+extern void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
+extern void ExplainCloseWorker(ExplainState *es);
+extern void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
+
+
#endif /* EXPLAIN_H */
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
new file mode 100644
index 0000000000..8034a4e0db
--- /dev/null
+++ b/src/test/regress/expected/explain.out
@@ -0,0 +1,114 @@
+--
+-- EXPLAIN
+--
+-- helper functions for examining explain output
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+ explain_workers
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ { "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Sort Method" : "quicksort", "Sort Space Used" : 0, "Sort Space Type" : "Memory", "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }], "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }] }] }] }
+(1 row)
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers: .+?$', 'Buffers: xxx', 'n'), 'Sort Method: .+?$', 'Sort Method: xxx', 'n');
+ regexp_replace
+-------------------------------------------------------------------
+ Gather (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Workers Planned: 1 +
+ Workers Launched: 1 +
+ Single Copy: true +
+ Buffers: xxx +
+ -> Sort (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Sort Key: (ROW("*VALUES*".column1)) +
+ Buffers: shared hit=114 +
+ Worker 0: actual rows=2 loops=1 +
+ Buffers: shared hit=114 +
+ Sort Method: xxx +
+ -> Values Scan on "*VALUES*" (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, ROW("*VALUES*".column1)+
+ Worker 0: actual rows=2 loops=1 +
+
+(1 row)
+
+reset force_parallel_mode;
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 96dfb7c8dd..cba15f7133 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -591,10 +591,10 @@ select * from explain_parallel_sort_stats();
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
Sort Method: quicksort Memory: xxx
- Worker 0: Sort Method: quicksort Memory: xxx
- Worker 1: Sort Method: quicksort Memory: xxx
- Worker 2: Sort Method: quicksort Memory: xxx
- Worker 3: Sort Method: quicksort Memory: xxx
+ Worker 0: Sort Method: quicksort Memory: xxx
+ Worker 1: Sort Method: quicksort Memory: xxx
+ Worker 2: Sort Method: quicksort Memory: xxx
+ Worker 3: Sort Method: quicksort Memory: xxx
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
(14 rows)
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index d33a4e143d..d2b17dd3ea 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,7 +112,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
# event triggers cannot run concurrently with any test that runs DDL
test: event_trigger
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index f86f5c5682..acba391332 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -193,6 +193,7 @@ test: indexing
test: partition_aggregate
test: partition_info
test: tuplesort
+test: explain
test: event_trigger
test: fast_default
test: stats
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
new file mode 100644
index 0000000000..03accb1b83
--- /dev/null
+++ b/src/test/regress/sql/explain.sql
@@ -0,0 +1,98 @@
+--
+-- EXPLAIN
+--
+
+-- helper functions for examining explain output
+
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+
+
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers: .+?$', 'Buffers: xxx', 'n'), 'Sort Method: .+?$', 'Sort Method: xxx', 'n');
+
+reset force_parallel_mode;
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-22 12:54 Georgios Kokolatos <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-22 12:54 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
> TEXT format was tricky due to its inconsistencies, but I think I have
> something working reasonably well. I added a simple test for TEXT
> format output as well, using a similar approach as the JSON format
Great!
> test, and liberally regexp_replacing away any volatile output. I
> suppose in theory we could do this for YAML, too, but I think it's
> gross enough not to be worth it, especially given the high similarity
> of all the structured outputs.
Agreed, what is in the patch suffices. Overall great work, a couple of
minor nitpicks if you allow me.
+ /* Prepare per-worker output */
+ if (es->analyze && planstate->worker_instrument) {
Style, parenthesis on its own line.
+ int num_workers = planstate->worker_instrument->num_workers;
+ int n;
+ worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
+ for (n = 0; n < num_workers; n++) {
I think C99 would be better here. Also no parenthesis needed.
+ worker_strs[n] = makeStringInfo();
+ }
+ }
@@ -1357,6 +1369,58 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyBool("Parallel Aware", plan->parallel_aware, es);
}
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && planstate->worker_instrument)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+ int n;
+
+ for (n = 0; n < w->num_workers; ++n)
I think C99 would be better here.
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ if (indent)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ }
Style: No parenthesis needed
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ /* Show worker detail */
+ if (planstate->worker_instrument) {
+ ExplainFlushWorkers(worker_strs, planstate->worker_instrument->num_workers, es);
}
Style: No parenthesis needed
+ * just indent once, to add worker info on the next worker line.
+ */
+ if (es->str == es->root_str)
+ {
+ es->indent += es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+ }
+
Style: No parenthesis needed
+ ExplainCloseGroup("Workers", "Workers", false, es);
+ // do we have any other cleanup to do?
This comment does not really explain anything. Either remove
or rephrase. Also C style comments.
+ es->print_workers = false;
+}
int indent; /* current indentation level */
List *grouping_stack; /* format-specific grouping state */
+ bool print_workers; /* whether current node has worker metadata */
Hmm.. commit <b925a00f4ef> introduced `hide_workers` in the struct. Having both
names in the struct so far apart even seems a bit confusing and sloppy. Do you
think it would be possible to combine or rename?
+extern void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
+extern void ExplainCloseWorker(ExplainState *es);
+extern void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
No need to expose those, is there? I feel there should be static.
Awaiting for answer or resolution of these comments to change the status.
//Georgios
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-22 16:54 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-22 16:54 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
Thanks! I'll fix the brace issues. Re: the other items:
> + int num_workers = planstate->worker_instrument->num_workers;
> + int n;
> + worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
> + for (n = 0; n < num_workers; n++) {
>
> I think C99 would be better here. Also no parenthesis needed.
Pardon my C illiteracy, but what am I doing that's not valid C99 here?
> + /* Prepare worker general execution details */
> + if (es->analyze && es->verbose && planstate->worker_instrument)
> + {
> + WorkerInstrumentation *w = planstate->worker_instrument;
> + int n;
> +
> + for (n = 0; n < w->num_workers; ++n)
>
> I think C99 would be better here.
And here (if it's not the same problem)?
> + ExplainCloseGroup("Workers", "Workers", false, es);
> + // do we have any other cleanup to do?
>
> This comment does not really explain anything. Either remove
> or rephrase. Also C style comments.
Good catch, thanks--I had put this in to remind myself (and reviewers)
about cleanup, but I don't think there's anything else to do, so I'll
just drop it.
> int indent; /* current indentation level */
> List *grouping_stack; /* format-specific grouping state */
> + bool print_workers; /* whether current node has worker metadata */
>
> Hmm.. commit <b925a00f4ef> introduced `hide_workers` in the struct. Having both
> names in the struct so far apart even seems a bit confusing and sloppy. Do you
> think it would be possible to combine or rename?
I noticed that. I was thinking about combining them, but
"hide_workers" seems to be about "pretend there is no worker output
even if there is" and "print_workers" is "keep track of whether or not
there is worker output to print". Maybe I'll rename to
"has_worker_output"?
> +extern void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
> +extern void ExplainCloseWorker(ExplainState *es);
> +extern void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
>
> No need to expose those, is there? I feel there should be static.
Good call, I'll update.
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-22 17:36 Georgios Kokolatos <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-22 17:36 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
>> + int num_workers = planstate->worker_instrument->num_workers;
>> + int n;
>> + worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
>> + for (n = 0; n < num_workers; n++) {
>>
>> I think C99 would be better here. Also no parenthesis needed.
>
>
> Pardon my C illiteracy, but what am I doing that's not valid C99 here?
My bad, I should have been more clear. I meant that it is preferable to use
the C99 standard which calls for declaring variables in the scope that you
need them. In this case, 'n' is needed only in the for loop, so something like
for (int n = 0; n < num_workers; n++)
is preferable. To be clear, your code was perfectly valid. It was only the
style I was referring to.
>> + for (n = 0; n < w->num_workers; ++n)
>>
>> I think C99 would be better here.
>
>
> And here (if it's not the same problem)?
Exactly the same as above.
>> int indent; /* current indentation level */
>> List *grouping_stack; /* format-specific grouping state */
>> + bool print_workers; /* whether current node has worker metadata */
>>
>> Hmm.. commit <b925a00f4ef> introduced `hide_workers` in the struct. Having both
>> names in the struct so far apart even seems a bit confusing and sloppy. Do you
>> think it would be possible to combine or rename?
>
>
> I noticed that. I was thinking about combining them, but
> "hide_workers" seems to be about "pretend there is no worker output
> even if there is" and "print_workers" is "keep track of whether or not
> there is worker output to print". Maybe I'll rename to
> "has_worker_output"?
The rename sounds a bit better in my humble opinion. Thanks.
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-23 09:00 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-23 09:00 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
On Wed, Jan 22, 2020 at 9:37 AM Georgios Kokolatos <[email protected]> wrote:
> My bad, I should have been more clear. I meant that it is preferable to use
> the C99 standard which calls for declaring variables in the scope that you
> need them.
Ah, I see. I think I got that from ExplainPrintSettings. I've
corrected my usage--thanks for pointing it out. I appreciate the
effort to maintain a consistent style.
>
> >> int indent; /* current indentation level */
> >> List *grouping_stack; /* format-specific grouping state */
> >> + bool print_workers; /* whether current node has worker metadata */
> >>
> >> Hmm.. commit <b925a00f4ef> introduced `hide_workers` in the struct. Having both
> >> names in the struct so far apart even seems a bit confusing and sloppy. Do you
> >> think it would be possible to combine or rename?
> >
> >
> > I noticed that. I was thinking about combining them, but
> > "hide_workers" seems to be about "pretend there is no worker output
> > even if there is" and "print_workers" is "keep track of whether or not
> > there is worker output to print". Maybe I'll rename to
> > "has_worker_output"?
>
> The rename sounds a bit better in my humble opinion. Thanks.
Also, reviewing my code again, I noticed that when I moved the general
worker output earlier, I missed part of the merge conflict: I had
replaced
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
with
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && planstate->worker_instrument)
which ignores the es->hide_workers flag (it did not fail the tests,
but the intent is pretty clear). I've corrected this in the current
patch.
I also noticed that we can now handle worker buffer output more
consistently across TEXT and structured formats, so I made that small
change too:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 140d0be426..b23b015594 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1401,8 +1401,6 @@ ExplainNode(PlanState *planstate, List *ancestors,
appendStringInfo(es->str,
"actual rows=%.0f loops=%.0f\n",
rows, nloops);
- if (es->buffers)
- show_buffer_usage(es,
&instrument->bufusage);
}
else
{
@@ -1951,7 +1949,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
/* Prepare worker buffer usage */
if (es->buffers && es->analyze && es->verbose && !es->hide_workers
- && planstate->worker_instrument && es->format !=
EXPLAIN_FORMAT_TEXT)
+ && planstate->worker_instrument)
{
WorkerInstrumentation *w = planstate->worker_instrument;
int n;
diff --git a/src/test/regress/expected/explain.out
b/src/test/regress/expected/explain.out
index 8034a4e0db..a4eed3067f 100644
--- a/src/test/regress/expected/explain.out
+++ b/src/test/regress/expected/explain.out
@@ -103,8 +103,8 @@ $$, 'verbose', 'analyze', 'buffers', 'timing off',
'costs off', 'summary off'),
Sort Key: (ROW("*VALUES*".column1)) +
Buffers: shared hit=114 +
Worker 0: actual rows=2 loops=1 +
- Buffers: shared hit=114 +
Sort Method: xxx +
+ Buffers: shared hit=114 +
-> Values Scan on "*VALUES*" (actual rows=2 loops=1) +
Output: "*VALUES*".column1, ROW("*VALUES*".column1)+
Worker 0: actual rows=2 loops=1 +
I think the "producing plan output for a worker" process is easier to
reason about now, and while it changes TEXT format worker output
order, the other changes in this patch are more drastic so this
probably does not matter.
I've also addressed the other feedback above, and reworded a couple of
comments slightly.
Attachments:
[text/x-patch] merge-explain-worker-output-v4.patch (23.2K, ../../CAOtHd0D+fHCSDsm1ChZxfpgi_Fsce121GgGw4gG2EmWd9sn1Gg@mail.gmail.com/2-merge-explain-worker-output-v4.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..b4108f82ec 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -100,7 +100,7 @@ static void show_sortorder_options(StringInfo buf, Node *sortexpr,
Oid sortOperator, Oid collation, bool nullsFirst);
static void show_tablesample(TableSampleClause *tsc, PlanState *planstate,
List *ancestors, ExplainState *es);
-static void show_sort_info(SortState *sortstate, ExplainState *es);
+static void show_sort_info(SortState *sortstate, StringInfo* worker_strs, ExplainState *es);
static void show_hash_info(HashState *hashstate, ExplainState *es);
static void show_tidbitmap_info(BitmapHeapScanState *planstate,
ExplainState *es);
@@ -131,7 +131,9 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
-
+static void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
+static void ExplainCloseWorker(ExplainState *es);
+static void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
/*
@@ -289,6 +291,7 @@ NewExplainState(void)
es->costs = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
+ es->root_str = es->str;
return es;
}
@@ -1090,9 +1093,19 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ StringInfo *worker_strs = NULL;
int save_indent = es->indent;
bool haschildren;
+ /* Prepare per-worker output */
+ if (es->analyze && planstate->worker_instrument)
+ {
+ int num_workers = planstate->worker_instrument->num_workers;
+ worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
+ for (int n = 0; n < num_workers; n++)
+ worker_strs[n] = makeStringInfo();
+ }
+
switch (nodeTag(plan))
{
case T_Result:
@@ -1357,6 +1370,55 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyBool("Parallel Aware", plan->parallel_aware, es);
}
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && !es->hide_workers && planstate->worker_instrument)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+
+ for (int n = 0; n < w->num_workers; ++n)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ ExplainOpenWorker(worker_strs[n], es);
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ }
+ else
+ {
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+ }
+
+ ExplainCloseWorker(es);
+ }
+
switch (nodeTag(plan))
{
case T_SeqScan:
@@ -1856,7 +1918,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
break;
case T_Sort:
show_sort_keys(castNode(SortState, planstate), ancestors, es);
- show_sort_info(castNode(SortState, planstate), es);
+ show_sort_info(castNode(SortState, planstate), worker_strs, es);
break;
case T_MergeAppend:
show_merge_append_keys(castNode(MergeAppendState, planstate),
@@ -1885,76 +1947,31 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare worker buffer usage */
+ if (es->buffers && es->analyze && es->verbose && !es->hide_workers
+ && planstate->worker_instrument)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
int n;
for (n = 0; n < w->num_workers; ++n)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
-
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
-
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(worker_strs[n], es);
+ show_buffer_usage(es, &instrument->bufusage);
}
-
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ ExplainCloseWorker(es);
}
+ /* Show worker detail */
+ if (planstate->worker_instrument)
+ ExplainFlushWorkers(worker_strs, planstate->worker_instrument->num_workers, es);
+
/* Get ready to display the child plans */
haschildren = planstate->initPlan ||
outerPlanState(planstate) ||
@@ -2552,7 +2569,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
* If it's EXPLAIN ANALYZE, show tuplesort stats for a sort node
*/
static void
-show_sort_info(SortState *sortstate, ExplainState *es)
+show_sort_info(SortState *sortstate, StringInfo *worker_strs, ExplainState *es)
{
if (!es->analyze)
return;
@@ -2593,7 +2610,7 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
+ bool indent = es->has_worker_output;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2609,32 +2626,24 @@ show_sort_info(SortState *sortstate, ExplainState *es)
spaceType = tuplesort_space_type_name(sinstrument->spaceType);
spaceUsed = sinstrument->spaceUsed;
+ ExplainOpenWorker(worker_strs[n], es);
+
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ if (indent)
+ appendStringInfoSpaces(es->str, es->indent * 2);
appendStringInfo(es->str,
"Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ ExplainCloseWorker(es);
}
}
@@ -3751,6 +3760,68 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
}
+/*
+ * Begin output for a specific worker. If we are currently producing output
+ * for another worker, close that worker automatically.
+ */
+static void
+ExplainOpenWorker(StringInfo worker_str, ExplainState *es)
+{
+ /*
+ * For structured formats, we indent twice: once for the "Worker" group
+ * itself and once for the surrounding "Workers" group. For text format,
+ * we just indent once, to add worker info on the next worker line. Either
+ * way, we only need to adjust indentation once for each string of
+ * ExplainOpenWorker calls.
+ */
+ if (es->str == es->root_str)
+ es->indent += es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+
+ es->has_worker_output = true;
+ es->str = worker_str;
+}
+
+/*
+ * Explicitly end output for the current worker and resume output
+ * for the root of the node.
+ */
+static void
+ExplainCloseWorker(ExplainState *es)
+{
+ /* as above, the text format has different indentation */
+ es->indent -= es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+
+ es->str = es->root_str;
+}
+
+/*
+ * Flush worker output for this node and reset the worker output state.
+ */
+static void
+ExplainFlushWorkers(StringInfo* worker_strs, int num_workers, ExplainState *es)
+{
+ if (!es->has_worker_output)
+ return;
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (int i = 0; i < num_workers; i++)
+ {
+ ExplainOpenGroup("Worker", NULL, true, es);
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ appendStringInfo(es->str, "Worker %d: ", i);
+ }
+ else
+ ExplainPropertyInteger("Worker Number", NULL, i, es);
+
+ appendStringInfoString(es->str, worker_strs[i]->data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+ es->has_worker_output = false;
+}
+
/*
* Emit a "dummy" group that never has any members.
*
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7c47..1f9208b571 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -27,7 +27,7 @@ typedef enum ExplainFormat
typedef struct ExplainState
{
- StringInfo str; /* output buffer */
+ StringInfo str; /* current output buffer */
/* options */
bool verbose; /* be verbose */
bool analyze; /* print actual times */
@@ -40,6 +40,8 @@ typedef struct ExplainState
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
List *grouping_stack; /* format-specific grouping state */
+ bool has_worker_output; /* whether current node has worker metadata */
+ StringInfo root_str; /* root output buffer */
/* state related to the current plan tree (filled by ExplainPrintPlan) */
PlannedStmt *pstmt; /* top of plan */
List *rtable; /* range table */
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
new file mode 100644
index 0000000000..a4eed3067f
--- /dev/null
+++ b/src/test/regress/expected/explain.out
@@ -0,0 +1,114 @@
+--
+-- EXPLAIN
+--
+-- helper functions for examining explain output
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+ explain_workers
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ { "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Sort Method" : "quicksort", "Sort Space Used" : 0, "Sort Space Type" : "Memory", "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }], "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }] }] }] }
+(1 row)
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers: .+?$', 'Buffers: xxx', 'n'), 'Sort Method: .+?$', 'Sort Method: xxx', 'n');
+ regexp_replace
+-------------------------------------------------------------------
+ Gather (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Workers Planned: 1 +
+ Workers Launched: 1 +
+ Single Copy: true +
+ Buffers: xxx +
+ -> Sort (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Sort Key: (ROW("*VALUES*".column1)) +
+ Buffers: shared hit=114 +
+ Worker 0: actual rows=2 loops=1 +
+ Sort Method: xxx +
+ Buffers: shared hit=114 +
+ -> Values Scan on "*VALUES*" (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, ROW("*VALUES*".column1)+
+ Worker 0: actual rows=2 loops=1 +
+
+(1 row)
+
+reset force_parallel_mode;
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 96dfb7c8dd..cba15f7133 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -591,10 +591,10 @@ select * from explain_parallel_sort_stats();
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
Sort Method: quicksort Memory: xxx
- Worker 0: Sort Method: quicksort Memory: xxx
- Worker 1: Sort Method: quicksort Memory: xxx
- Worker 2: Sort Method: quicksort Memory: xxx
- Worker 3: Sort Method: quicksort Memory: xxx
+ Worker 0: Sort Method: quicksort Memory: xxx
+ Worker 1: Sort Method: quicksort Memory: xxx
+ Worker 2: Sort Method: quicksort Memory: xxx
+ Worker 3: Sort Method: quicksort Memory: xxx
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
(14 rows)
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index d33a4e143d..d2b17dd3ea 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,7 +112,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
# event triggers cannot run concurrently with any test that runs DDL
test: event_trigger
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index f86f5c5682..acba391332 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -193,6 +193,7 @@ test: indexing
test: partition_aggregate
test: partition_info
test: tuplesort
+test: explain
test: event_trigger
test: fast_default
test: stats
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
new file mode 100644
index 0000000000..03accb1b83
--- /dev/null
+++ b/src/test/regress/sql/explain.sql
@@ -0,0 +1,98 @@
+--
+-- EXPLAIN
+--
+
+-- helper functions for examining explain output
+
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+
+
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers: .+?$', 'Buffers: xxx', 'n'), 'Sort Method: .+?$', 'Sort Method: xxx', 'n');
+
+reset force_parallel_mode;
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-23 10:38 Georgios Kokolatos <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Georgios Kokolatos @ 2020-01-23 10:38 UTC (permalink / raw)
To: [email protected]; +Cc: Maciek Sakrejda <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
> Ah, I see. I think I got that from ExplainPrintSettings. I've
> corrected my usage--thanks for pointing it out. I appreciate the
> effort to maintain a consistent style.
Thanks, I am just following the reviewing guide to be honest.
> Also, reviewing my code again, I noticed that when I moved the general
> worker output earlier, I missed part of the merge conflict:
Right. I thought that was intentional.
> which ignores the es->hide_workers flag (it did not fail the tests,
> but the intent is pretty clear). I've corrected this in the current
> patch.
Noted and appreciated.
> I also noticed that we can now handle worker buffer output more
> consistently across TEXT and structured formats, so I made that small
> change too:
Looks good.
> I think the "producing plan output for a worker" process is easier to
> reason about now, and while it changes TEXT format worker output
> order, the other changes in this patch are more drastic so this
> probably does not matter.
>
> I've also addressed the other feedback above, and reworded a couple of
> comments slightly.
Thanks for the above. Looks clean, does what it says in the tin and solves a
problem that needs solving. All relevant installcheck-world pass. As far as I
am concerned, I think it is ready to be sent to a committer. Updating the status
accordingly.
The new status of this patch is: Ready for Committer
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-24 17:03 Maciek Sakrejda <[email protected]>
parent: Georgios Kokolatos <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-24 17:03 UTC (permalink / raw)
To: Georgios Kokolatos <[email protected]>; +Cc: [email protected]
Great, thank you. I noticed in the CF patch tester app, the build
fails on Windows [1]. Investigating, I realized I had failed to fully
strip volatile EXPLAIN info (namely buffers) in TEXT mode due to a
bad regexp_replace. I've fixed this in the attached patch (which I
also rebased against current master again).
[1]: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.76313
Attachments:
[text/x-patch] merge-explain-worker-output-v5.patch (23.2K, ../../CAOtHd0BksSjoSPd5GfdOxZh87Cgs3_2WEOQf2ec3pObMuW+oDg@mail.gmail.com/2-merge-explain-worker-output-v5.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d573..b4108f82ec 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -100,7 +100,7 @@ static void show_sortorder_options(StringInfo buf, Node *sortexpr,
Oid sortOperator, Oid collation, bool nullsFirst);
static void show_tablesample(TableSampleClause *tsc, PlanState *planstate,
List *ancestors, ExplainState *es);
-static void show_sort_info(SortState *sortstate, ExplainState *es);
+static void show_sort_info(SortState *sortstate, StringInfo* worker_strs, ExplainState *es);
static void show_hash_info(HashState *hashstate, ExplainState *es);
static void show_tidbitmap_info(BitmapHeapScanState *planstate,
ExplainState *es);
@@ -131,7 +131,9 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
-
+static void ExplainOpenWorker(StringInfo worker_str, ExplainState *es);
+static void ExplainCloseWorker(ExplainState *es);
+static void ExplainFlushWorkers(StringInfo *worker_strs, int num_workers, ExplainState *es);
/*
@@ -289,6 +291,7 @@ NewExplainState(void)
es->costs = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
+ es->root_str = es->str;
return es;
}
@@ -1090,9 +1093,19 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ StringInfo *worker_strs = NULL;
int save_indent = es->indent;
bool haschildren;
+ /* Prepare per-worker output */
+ if (es->analyze && planstate->worker_instrument)
+ {
+ int num_workers = planstate->worker_instrument->num_workers;
+ worker_strs = (StringInfo *) palloc0(num_workers * sizeof(StringInfo));
+ for (int n = 0; n < num_workers; n++)
+ worker_strs[n] = makeStringInfo();
+ }
+
switch (nodeTag(plan))
{
case T_Result:
@@ -1357,6 +1370,55 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyBool("Parallel Aware", plan->parallel_aware, es);
}
+ /* Prepare worker general execution details */
+ if (es->analyze && es->verbose && !es->hide_workers && planstate->worker_instrument)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+
+ for (int n = 0; n < w->num_workers; ++n)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ ExplainOpenWorker(worker_strs[n], es);
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ }
+ else
+ {
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+ }
+
+ ExplainCloseWorker(es);
+ }
+
switch (nodeTag(plan))
{
case T_SeqScan:
@@ -1856,7 +1918,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
break;
case T_Sort:
show_sort_keys(castNode(SortState, planstate), ancestors, es);
- show_sort_info(castNode(SortState, planstate), es);
+ show_sort_info(castNode(SortState, planstate), worker_strs, es);
break;
case T_MergeAppend:
show_merge_append_keys(castNode(MergeAppendState, planstate),
@@ -1885,76 +1947,31 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare worker buffer usage */
+ if (es->buffers && es->analyze && es->verbose && !es->hide_workers
+ && planstate->worker_instrument)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
int n;
for (n = 0; n < w->num_workers; ++n)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
-
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
-
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(worker_strs[n], es);
+ show_buffer_usage(es, &instrument->bufusage);
}
-
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ ExplainCloseWorker(es);
}
+ /* Show worker detail */
+ if (planstate->worker_instrument)
+ ExplainFlushWorkers(worker_strs, planstate->worker_instrument->num_workers, es);
+
/* Get ready to display the child plans */
haschildren = planstate->initPlan ||
outerPlanState(planstate) ||
@@ -2552,7 +2569,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
* If it's EXPLAIN ANALYZE, show tuplesort stats for a sort node
*/
static void
-show_sort_info(SortState *sortstate, ExplainState *es)
+show_sort_info(SortState *sortstate, StringInfo *worker_strs, ExplainState *es)
{
if (!es->analyze)
return;
@@ -2593,7 +2610,7 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
+ bool indent = es->has_worker_output;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2609,32 +2626,24 @@ show_sort_info(SortState *sortstate, ExplainState *es)
spaceType = tuplesort_space_type_name(sinstrument->spaceType);
spaceUsed = sinstrument->spaceUsed;
+ ExplainOpenWorker(worker_strs[n], es);
+
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ if (indent)
+ appendStringInfoSpaces(es->str, es->indent * 2);
appendStringInfo(es->str,
"Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
+ ExplainCloseWorker(es);
}
}
@@ -3751,6 +3760,68 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
}
+/*
+ * Begin output for a specific worker. If we are currently producing output
+ * for another worker, close that worker automatically.
+ */
+static void
+ExplainOpenWorker(StringInfo worker_str, ExplainState *es)
+{
+ /*
+ * For structured formats, we indent twice: once for the "Worker" group
+ * itself and once for the surrounding "Workers" group. For text format,
+ * we just indent once, to add worker info on the next worker line. Either
+ * way, we only need to adjust indentation once for each string of
+ * ExplainOpenWorker calls.
+ */
+ if (es->str == es->root_str)
+ es->indent += es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+
+ es->has_worker_output = true;
+ es->str = worker_str;
+}
+
+/*
+ * Explicitly end output for the current worker and resume output
+ * for the root of the node.
+ */
+static void
+ExplainCloseWorker(ExplainState *es)
+{
+ /* as above, the text format has different indentation */
+ es->indent -= es->format == EXPLAIN_FORMAT_TEXT ? 1 : 2;
+
+ es->str = es->root_str;
+}
+
+/*
+ * Flush worker output for this node and reset the worker output state.
+ */
+static void
+ExplainFlushWorkers(StringInfo* worker_strs, int num_workers, ExplainState *es)
+{
+ if (!es->has_worker_output)
+ return;
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (int i = 0; i < num_workers; i++)
+ {
+ ExplainOpenGroup("Worker", NULL, true, es);
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ appendStringInfoSpaces(es->str, es->indent * 2);
+ appendStringInfo(es->str, "Worker %d: ", i);
+ }
+ else
+ ExplainPropertyInteger("Worker Number", NULL, i, es);
+
+ appendStringInfoString(es->str, worker_strs[i]->data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+ es->has_worker_output = false;
+}
+
/*
* Emit a "dummy" group that never has any members.
*
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7c47..1f9208b571 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -27,7 +27,7 @@ typedef enum ExplainFormat
typedef struct ExplainState
{
- StringInfo str; /* output buffer */
+ StringInfo str; /* current output buffer */
/* options */
bool verbose; /* be verbose */
bool analyze; /* print actual times */
@@ -40,6 +40,8 @@ typedef struct ExplainState
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
List *grouping_stack; /* format-specific grouping state */
+ bool has_worker_output; /* whether current node has worker metadata */
+ StringInfo root_str; /* root output buffer */
/* state related to the current plan tree (filled by ExplainPrintPlan) */
PlannedStmt *pstmt; /* top of plan */
List *rtable; /* range table */
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
new file mode 100644
index 0000000000..12ea1f0e5b
--- /dev/null
+++ b/src/test/regress/expected/explain.out
@@ -0,0 +1,114 @@
+--
+-- EXPLAIN
+--
+-- helper functions for examining explain output
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+ explain_workers
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ { "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Sort Method" : "quicksort", "Sort Space Used" : 0, "Sort Space Type" : "Memory", "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }], "Plans" : [{ "Workers" : [{ "Worker Number" : 0, "Actual Startup Time" : 0, "Actual Total Time" : 0, "Actual Rows" : 2, "Actual Loops" : 1, "Shared Hit Blocks" : 0, "Shared Read Blocks" : 0, "Shared Dirtied Blocks" : 0, "Shared Written Blocks" : 0, "Local Hit Blocks" : 0, "Local Read Blocks" : 0, "Local Dirtied Blocks" : 0, "Local Written Blocks" : 0, "Temp Read Blocks" : 0, "Temp Written Blocks" : 0 }] }] }] }
+(1 row)
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers:.+$', 'Buffers: xxx', 'ng'), 'Sort Method:.+$', 'Sort Method: xxx', 'ng');
+ regexp_replace
+-------------------------------------------------------------------
+ Gather (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Workers Planned: 1 +
+ Workers Launched: 1 +
+ Single Copy: true +
+ Buffers: xxx +
+ -> Sort (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, (ROW("*VALUES*".column1)) +
+ Sort Key: (ROW("*VALUES*".column1)) +
+ Buffers: xxx +
+ Worker 0: actual rows=2 loops=1 +
+ Sort Method: xxx +
+ Buffers: xxx +
+ -> Values Scan on "*VALUES*" (actual rows=2 loops=1) +
+ Output: "*VALUES*".column1, ROW("*VALUES*".column1)+
+ Worker 0: actual rows=2 loops=1 +
+
+(1 row)
+
+reset force_parallel_mode;
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 96dfb7c8dd..cba15f7133 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -591,10 +591,10 @@ select * from explain_parallel_sort_stats();
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
Sort Method: quicksort Memory: xxx
- Worker 0: Sort Method: quicksort Memory: xxx
- Worker 1: Sort Method: quicksort Memory: xxx
- Worker 2: Sort Method: quicksort Memory: xxx
- Worker 3: Sort Method: quicksort Memory: xxx
+ Worker 0: Sort Method: quicksort Memory: xxx
+ Worker 1: Sort Method: quicksort Memory: xxx
+ Worker 2: Sort Method: quicksort Memory: xxx
+ Worker 3: Sort Method: quicksort Memory: xxx
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
(14 rows)
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index d33a4e143d..d2b17dd3ea 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -112,7 +112,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
# event triggers cannot run concurrently with any test that runs DDL
test: event_trigger
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index f86f5c5682..acba391332 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -193,6 +193,7 @@ test: indexing
test: partition_aggregate
test: partition_info
test: tuplesort
+test: explain
test: event_trigger
test: fast_default
test: stats
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
new file mode 100644
index 0000000000..af443000da
--- /dev/null
+++ b/src/test/regress/sql/explain.sql
@@ -0,0 +1,98 @@
+--
+-- EXPLAIN
+--
+
+-- helper functions for examining explain output
+
+-- return a json explain of given query
+CREATE OR REPLACE FUNCTION json_explain(query text, variadic opts text[])
+ RETURNS json LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan json;
+BEGIN
+ EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT JSON) '), ', ') || query INTO STRICT plan;
+ RETURN plan;
+END;
+$$;
+
+-- return a text explain of given query
+CREATE OR REPLACE FUNCTION text_explain(query text, variadic opts text[])
+ RETURNS text LANGUAGE PLPGSQL AS $$
+DECLARE
+ plan_row RECORD;
+ plan text = '';
+BEGIN
+ FOR plan_row IN EXECUTE 'EXPLAIN (' || array_to_string(array_append(opts, 'FORMAT TEXT) '), ', ') || query LOOP
+ plan = plan || plan_row."QUERY PLAN" || E'\n';
+ END LOOP;
+
+ RETURN plan;
+END;
+$$;
+
+
+/*
+ * Takes a json object and processes its keys and values. For every
+ * matching key whose value is a number, set it to zero to facilitate
+ * comparisons for values that can vary across executions. If not a number,
+ * set the value to -1 to indicate we matched an unexpected key. Return
+ * non-matching keys and values unmodified.
+ */
+CREATE OR REPLACE FUNCTION json_normalize_numeric_keys_matching(obj json, regexp text)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(
+ key,
+ CASE
+ WHEN key ~ regexp THEN
+ CASE json_typeof(value)
+ WHEN 'number' THEN 0::text::json
+ ELSE (-1)::text::json
+ END
+ ELSE value
+ END
+ ), '{}')
+FROM
+ json_each(obj)
+$$;
+
+-- explain only worker information, normalizing any values that could vary across executions
+CREATE OR REPLACE FUNCTION explain_workers(plan_node json)
+ RETURNS json LANGUAGE SQL AS $$
+SELECT
+ coalesce(json_object_agg(key, value), '{}')
+FROM (
+ SELECT
+ key,
+ CASE key
+ WHEN 'Workers' THEN
+ (SELECT json_agg(
+ json_normalize_numeric_keys_matching(worker, '(Blocks|Time)$|Sort Space Used')
+ )
+ FROM json_array_elements(value) AS workers(worker))
+ WHEN 'Plans' THEN
+ (SELECT json_agg(
+ explain_workers(child)
+ )
+ FROM json_array_elements(value) AS children(child))
+ END AS value
+ FROM
+ json_each(plan_node) AS entries(key, value)
+ WHERE
+ key IN ('Workers', 'Plans')
+) AS plan_fields(key, value);
+$$;
+
+-- test that per-worker sort and buffers output is combined correctly in EXPLAIN
+set force_parallel_mode = true;
+
+SELECT explain_workers(json_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers') -> 0 -> 'Plan');
+
+SELECT regexp_replace(regexp_replace(text_explain($$
+ SELECT * FROM (VALUES(1),(2)) x ORDER BY x;
+$$, 'verbose', 'analyze', 'buffers', 'timing off', 'costs off', 'summary off'),
+ 'Buffers:.+$', 'Buffers: xxx', 'ng'), 'Sort Method:.+$', 'Sort Method: xxx', 'ng');
+
+reset force_parallel_mode;
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-24 17:40 Tom Lane <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Tom Lane @ 2020-01-24 17:40 UTC (permalink / raw)
To: Maciek Sakrejda <[email protected]>; +Cc: Georgios Kokolatos <[email protected]>; [email protected]
Maciek Sakrejda <[email protected]> writes:
> Great, thank you. I noticed in the CF patch tester app, the build
> fails on Windows [1]. Investigating, I realized I had failed to fully
> strip volatile EXPLAIN info (namely buffers) in TEXT mode due to a
> bad regexp_replace.
You haven't gone nearly far enough in that direction. The proposed
output still relies on the assumption that the session will always
get as many workers as it asks for, which it will not. For previous
bitter experience in this department, see for instance commits 4ea03f3f4
and 13e8b2ee8.
TBH I am not sure that the proposed regression tests for this change
can be committed at all. Which is a bit of a problem perhaps, but
then again we don't have terribly good coverage for the existing code
either, for the same reason.
regards, tom lane
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-25 00:26 Maciek Sakrejda <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-25 00:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Georgios Kokolatos <[email protected]>; [email protected]
Okay. Does not getting as many workers as it asks for include
sometimes getting zero, completely changing the actual output? If so,
I'll submit a new version of the patch removing all tests--I was
hoping to improve coverage, but I guess this is not the way to start.
If not, can we keep the json tests at least if we only consider the
first worker?
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-25 02:14 Tom Lane <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Tom Lane @ 2020-01-25 02:14 UTC (permalink / raw)
To: Maciek Sakrejda <[email protected]>; +Cc: Georgios Kokolatos <[email protected]>; [email protected]
Maciek Sakrejda <[email protected]> writes:
> Okay. Does not getting as many workers as it asks for include
> sometimes getting zero, completely changing the actual output?
Yup :-(. We could possibly avoid that by running the explain
test by itself rather than in a parallel group, but I don't
especially want to add yet more non-parallelizable tests.
Meanwhile, I spent some time looking at the code, and wasn't very happy
with it. I'm on board with the general plan of redirecting EXPLAIN
output into per-worker buffers that we eventually recombine, but I think
that this particular coding is pretty unmaintainable/unextensible.
In particular, I'm really unhappy that the code is ignoring EXPLAIN's
grouping_state stack. That means that when it's formatting fields that
belong to the per-worker group, it's using the state-stack entry that
corresponds to the plan node's main level. This seems to accidentally
work, but that fact depends on a number of shaky assumptions:
* Before messing with any per-worker data, we've always emitted at
least one field in the plan node's main level, so that the state-stack
entry isn't at its initial state for the level.
* Before actually emitting the shunted-aside data, we've always emitted
a "Worker Number" field in correct format within the per-worker group,
so that the formatting state is now correct for a non-initial field.
* There is no formatting difference between any non-first fields in
a level (ie the state stack entries are basically just booleans),
so that it doesn't matter how many plan-node fields we emitted before
starting the per-worker data, so long as there was at least one, nor
does transiently abusing the plan node level's stack entry like this
break the formatting of subsequent plan-node-level fields.
Now maybe we'll never implement an output format that breaks that
last assumption, and maybe we'll never rearrange the EXPLAIN code
in a way that breaks either of the first two. But I don't like those
assumptions too much. I also didn't like the code's assumption that
all the non-text formats interpret es->indent the same.
I also noted an actual bug, which is that the patch fails regression
testing under force_parallel_mode = regress. This isn't really your
fault, because the issue is in this obscure and poorly-commented hack
in show_sort_info:
* You might think we should just skip this stanza entirely when
* es->hide_workers is true, but then we'd get no sort-method output at
* all. We have to make it look like worker 0's data is top-level data.
* Currently, we only bother with that for text-format output.
Nonetheless, it's broken.
So I spent some time hacking on this and came up with the attached.
It's noticeably more verbose than your patch, but it keeps the
output-format-aware code at what seems to me to be a maintainable
arm's-length distance from the parallel-worker hacking. TEXT is
still a special case of course :-(.
This patch just covers the code, I'm not taking any position yet
about what to do about the tests. I did tweak the code to eliminate
the one formatting difference in select_parallel (ie put two spaces
after "Worker N:", which I think reads better anyhow), so it
passes check-world as it stands.
Thoughts?
regards, tom lane
Attachments:
[text/x-diff] merge-explain-worker-output-v6.patch (23.2K, ../../[email protected]/2-merge-explain-worker-output-v6.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d..9e965d2 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -123,11 +123,20 @@ static void ExplainSubPlans(List *plans, List *ancestors,
const char *relationship, ExplainState *es);
static void ExplainCustomChildren(CustomScanState *css,
List *ancestors, ExplainState *es);
+static ExplainWorkersState *ExplainCreateWorkersState(int num_workers);
+static void ExplainOpenWorker(int n, ExplainState *es);
+static void ExplainCloseWorker(int n, ExplainState *es);
+static void ExplainFlushWorkersState(ExplainState *es);
static void ExplainProperty(const char *qlabel, const char *unit,
const char *value, bool numeric, ExplainState *es);
+static void ExplainOpenSetAsideGroup(const char *objtype, const char *labelname,
+ bool labeled, int depth, ExplainState *es);
+static void ExplainSaveGroup(ExplainState *es, int depth, int *state_save);
+static void ExplainRestoreGroup(ExplainState *es, int depth, int *state_save);
static void ExplainDummyGroup(const char *objtype, const char *labelname,
ExplainState *es);
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
+static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -827,7 +836,7 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
/* for higher density, open code the text output format */
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
if (for_workers)
appendStringInfo(es->str, "JIT for worker %u:\n", worker_num);
else
@@ -836,7 +845,7 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
ExplainPropertyInteger("Functions", NULL, ji->created_functions, es);
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Options: %s %s, %s %s, %s %s, %s %s\n",
"Inlining", jit_flags & PGJIT_INLINE ? "true" : "false",
"Optimization", jit_flags & PGJIT_OPT3 ? "true" : "false",
@@ -845,7 +854,7 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
if (es->analyze && es->timing)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Timing: %s %.3f ms, %s %.3f ms, %s %.3f ms, %s %.3f ms, %s %.3f ms\n",
"Generation", 1000.0 * INSTR_TIME_GET_DOUBLE(ji->generation_counter),
@@ -1074,9 +1083,10 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
* optional name to be attached to the node.
*
* In text format, es->indent is controlled in this function since we only
- * want it to change at plan-node boundaries. In non-text formats, es->indent
- * corresponds to the nesting depth of logical output groups, and therefore
- * is controlled by ExplainOpenGroup/ExplainCloseGroup.
+ * want it to change at plan-node boundaries (but a few subroutines will
+ * transiently increment it). In non-text formats, es->indent corresponds
+ * to the nesting depth of logical output groups, and therefore is controlled
+ * by ExplainOpenGroup/ExplainCloseGroup.
*/
static void
ExplainNode(PlanState *planstate, List *ancestors,
@@ -1090,9 +1100,20 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ ExplainWorkersState *save_workers_state = es->workers_state;
int save_indent = es->indent;
bool haschildren;
+ /*
+ * Prepare per-worker output buffers, if needed. We'll append the data in
+ * these to the main output string further down.
+ */
+ if (planstate->worker_instrument && es->analyze && !es->hide_workers)
+ es->workers_state = ExplainCreateWorkersState(planstate->worker_instrument->num_workers);
+ else
+ es->workers_state = NULL;
+
+ /* Identify plan node type, and print generic details */
switch (nodeTag(plan))
{
case T_Result:
@@ -1324,13 +1345,13 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
if (plan_name)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "%s\n", plan_name);
es->indent++;
}
if (es->indent)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "-> ");
es->indent += 2;
}
@@ -1574,6 +1595,56 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->format == EXPLAIN_FORMAT_TEXT)
appendStringInfoChar(es->str, '\n');
+ /* prepare per-worker general execution details */
+ if (es->workers_state && es->verbose)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+
+ for (int n = 0; n < w->num_workers; n++)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ ExplainOpenWorker(n, es);
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ ExplainIndentText(es);
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ }
+ else
+ {
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+
+ ExplainCloseWorker(n, es);
+ }
+ }
+
/* target list */
if (es->verbose)
show_plan_tlist(planstate, ancestors, es);
@@ -1885,76 +1956,30 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare per-worker buffer usage */
+ if (es->workers_state && es->buffers && es->verbose)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
- int n;
- for (n = 0; n < w->num_workers; ++n)
+ for (int n = 0; n < w->num_workers; n++)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
-
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
-
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(n, es);
+ show_buffer_usage(es, &instrument->bufusage);
+ ExplainCloseWorker(n, es);
}
-
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
}
+ /* Show per-worker details, then pop that stack */
+ if (es->workers_state)
+ ExplainFlushWorkersState(es);
+ es->workers_state = save_workers_state;
+
/* Get ready to display the child plans */
haschildren = planstate->initPlan ||
outerPlanState(planstate) ||
@@ -2525,7 +2550,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
{
bool first = true;
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Sampling: %s (", method_name);
foreach(lc, params)
{
@@ -2572,7 +2597,7 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
@@ -2588,12 +2613,14 @@ show_sort_info(SortState *sortstate, ExplainState *es)
* You might think we should just skip this stanza entirely when
* es->hide_workers is true, but then we'd get no sort-method output at
* all. We have to make it look like worker 0's data is top-level data.
- * Currently, we only bother with that for text-format output.
+ * This is easily done by just skipping the OpenWorker/CloseWorker calls.
+ * Currently, we don't worry about the possibility that there are multiple
+ * workers in such a case; if there are, duplicate output fields will be
+ * emitted.
*/
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2609,32 +2636,26 @@ show_sort_info(SortState *sortstate, ExplainState *es)
spaceType = tuplesort_space_type_name(sinstrument->spaceType);
spaceUsed = sinstrument->spaceUsed;
+ if (es->workers_state)
+ ExplainOpenWorker(n, es);
+
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
+
+ if (es->workers_state)
+ ExplainCloseWorker(n, es);
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
}
}
@@ -2721,7 +2742,7 @@ show_hash_info(HashState *hashstate, ExplainState *es)
else if (hinstrument.nbatch_original != hinstrument.nbatch ||
hinstrument.nbuckets_original != hinstrument.nbuckets)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Buckets: %d (originally %d) Batches: %d (originally %d) Memory Usage: %ldkB\n",
hinstrument.nbuckets,
@@ -2732,7 +2753,7 @@ show_hash_info(HashState *hashstate, ExplainState *es)
}
else
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Buckets: %d Batches: %d Memory Usage: %ldkB\n",
hinstrument.nbuckets, hinstrument.nbatch,
@@ -2758,7 +2779,7 @@ show_tidbitmap_info(BitmapHeapScanState *planstate, ExplainState *es)
{
if (planstate->exact_pages > 0 || planstate->lossy_pages > 0)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "Heap Blocks:");
if (planstate->exact_pages > 0)
appendStringInfo(es->str, " exact=%ld", planstate->exact_pages);
@@ -2894,7 +2915,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage)
/* Show only positive counter values. */
if (has_shared || has_local || has_temp)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "Buffers:");
if (has_shared)
@@ -2949,7 +2970,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage)
/* As above, show only positive counter values. */
if (has_timing)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "I/O Timings:");
if (!INSTR_TIME_IS_ZERO(usage->blk_read_time))
appendStringInfo(es->str, " read=%0.3f",
@@ -3237,7 +3258,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
*/
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str,
fdwroutine ? foperation : operation);
}
@@ -3427,6 +3448,150 @@ ExplainCustomChildren(CustomScanState *css, List *ancestors, ExplainState *es)
}
/*
+ * Create a per-plan-node workspace for collecting per-worker data.
+ */
+static ExplainWorkersState *
+ExplainCreateWorkersState(int num_workers)
+{
+ ExplainWorkersState *wstate;
+
+ wstate = (ExplainWorkersState *) palloc(sizeof(ExplainWorkersState));
+ wstate->num_workers = num_workers;
+ wstate->worker_inited = (bool *) palloc0(num_workers * sizeof(bool));
+ wstate->worker_str = (StringInfoData *)
+ palloc0(num_workers * sizeof(StringInfoData));
+ wstate->worker_state_save = (int *) palloc(num_workers * sizeof(int));
+ return wstate;
+}
+
+/*
+ * Begin or resume output into the set-aside group for worker N.
+ */
+static void
+ExplainOpenWorker(int n, ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ Assert(wstate);
+ Assert(n >= 0 && n < wstate->num_workers);
+
+ /* Save prior output pointer */
+ wstate->prev_str = es->str;
+
+ if (!wstate->worker_inited[n])
+ {
+ /* First time through, so create the buffer for this worker */
+ initStringInfo(&wstate->worker_str[n]);
+ es->str = &wstate->worker_str[n];
+
+ /*
+ * Push suitable initial formatting state for this worker's field
+ * group. We allow one extra logical nesting level, since this group
+ * will eventually be wrapped in an outer "Workers" group.
+ */
+ ExplainOpenSetAsideGroup("Worker", NULL, true, 2, es);
+
+ /*
+ * In non-TEXT formats, we always emit a "Worker Number" field even if
+ * there's no other data for this worker.
+ */
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Worker Number", NULL, n, es);
+
+ wstate->worker_inited[n] = true;
+ }
+ else
+ {
+ /* Resuming output for a worker we've already emitted some data for */
+ es->str = &wstate->worker_str[n];
+
+ /* Restore formatting state saved by last ExplainCloseWorker() */
+ ExplainRestoreGroup(es, 2, &wstate->worker_state_save[n]);
+ }
+
+ /*
+ * In TEXT format, prefix the first output line for this worker with
+ * "Worker N:". Then, any additional lines should be indented one more
+ * stop than the "Worker N" line is.
+ */
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ if (es->str->len == 0)
+ {
+ ExplainIndentText(es);
+ appendStringInfo(es->str, "Worker %d: ", n);
+ }
+
+ es->indent++;
+ }
+}
+
+/*
+ * End output for worker N --- must pair with previous ExplainOpenWorker call
+ */
+static void
+ExplainCloseWorker(int n, ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ Assert(wstate);
+ Assert(n >= 0 && n < wstate->num_workers);
+ Assert(wstate->worker_inited[n]);
+
+ /*
+ * Save formatting state in case we do another ExplainOpenWorker(), then
+ * pop the formatting stack.
+ */
+ ExplainSaveGroup(es, 2, &wstate->worker_state_save[n]);
+
+ /*
+ * In TEXT format, if we didn't actually produce any output line(s) then
+ * truncate off the partial line (this avoids bogus output if, for
+ * instance, show_buffer_usage chooses not to print anything). Also fix
+ * up the indent level.
+ */
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ while (es->str->len > 0 && es->str->data[es->str->len - 1] != '\n')
+ es->str->data[--(es->str->len)] = '\0';
+
+ es->indent--;
+ }
+
+ /* Restore prior output pointer */
+ es->str = wstate->prev_str;
+}
+
+/*
+ * Print per-worker info for current node, then free the ExplainWorkersState.
+ */
+static void
+ExplainFlushWorkersState(ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (int i = 0; i < wstate->num_workers; i++)
+ {
+ if (wstate->worker_inited[i])
+ {
+ /* This must match previous ExplainOpenSetAsideGroup call */
+ ExplainOpenGroup("Worker", NULL, true, es);
+ appendStringInfoString(es->str, wstate->worker_str[i].data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+
+ pfree(wstate->worker_str[i].data);
+ }
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+
+ pfree(wstate->worker_inited);
+ pfree(wstate->worker_str);
+ pfree(wstate->worker_state_save);
+ pfree(wstate);
+}
+
+/*
* Explain a property, such as sort keys or targets, that takes the form of
* a list of unlabeled items. "data" is a list of C strings.
*/
@@ -3439,7 +3604,7 @@ ExplainPropertyList(const char *qlabel, List *data, ExplainState *es)
switch (es->format)
{
case EXPLAIN_FORMAT_TEXT:
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "%s: ", qlabel);
foreach(lc, data)
{
@@ -3560,7 +3725,7 @@ ExplainProperty(const char *qlabel, const char *unit, const char *value,
switch (es->format)
{
case EXPLAIN_FORMAT_TEXT:
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
if (unit)
appendStringInfo(es->str, "%s: %s %s\n", qlabel, value, unit);
else
@@ -3752,6 +3917,117 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
/*
+ * Open a group of related objects, without emitting actual data.
+ *
+ * Prepare the formatting state as though we were beginning a group with
+ * the identified properties, but don't actually emit anything. Output
+ * subsequent to this call can be redirected into a separate output buffer,
+ * and then eventually appended to the main output buffer after doing a
+ * regular ExplainOpenGroup call (with the same parameters).
+ *
+ * The extra "depth" parameter is the new group's depth compared to current.
+ * It could be more than one, in case the eventual output will be enclosed
+ * in additional nesting group levels. We assume we don't need to track
+ * formatting state for those levels while preparing this group's output.
+ *
+ * There is no ExplainCloseSetAsideGroup --- in current usage, we always
+ * pop this state with ExplainSaveGroup.
+ */
+static void
+ExplainOpenSetAsideGroup(const char *objtype, const char *labelname,
+ bool labeled, int depth, ExplainState *es)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->grouping_stack = lcons_int(0, es->grouping_stack);
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ if (labelname)
+ es->grouping_stack = lcons_int(1, es->grouping_stack);
+ else
+ es->grouping_stack = lcons_int(0, es->grouping_stack);
+ es->indent += depth;
+ break;
+ }
+}
+
+/*
+ * Pop one level of grouping state, allowing for a re-push later.
+ *
+ * This is typically used after ExplainOpenSetAsideGroup; pass the
+ * same "depth" used for that.
+ *
+ * This should not emit any output. If state needs to be saved,
+ * save it at *state_save. Currently, an integer save area is sufficient
+ * for all formats, but we might need to revisit that someday.
+ */
+static void
+ExplainSaveGroup(ExplainState *es, int depth, int *state_save)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent -= depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->indent -= depth;
+ *state_save = linitial_int(es->grouping_stack);
+ es->grouping_stack = list_delete_first(es->grouping_stack);
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ es->indent -= depth;
+ *state_save = linitial_int(es->grouping_stack);
+ es->grouping_stack = list_delete_first(es->grouping_stack);
+ break;
+ }
+}
+
+/*
+ * Re-push one level of grouping state, undoing the effects of ExplainSaveGroup.
+ */
+static void
+ExplainRestoreGroup(ExplainState *es, int depth, int *state_save)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->grouping_stack = lcons_int(*state_save, es->grouping_stack);
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ es->grouping_stack = lcons_int(*state_save, es->grouping_stack);
+ es->indent += depth;
+ break;
+ }
+}
+
+/*
* Emit a "dummy" group that never has any members.
*
* objtype is the type of the group object, labelname is its label within
@@ -3913,6 +4189,21 @@ ExplainXMLTag(const char *tagname, int flags, ExplainState *es)
}
/*
+ * Indent a text-format line.
+ *
+ * We indent by two spaces per indentation level. However, when emitting
+ * data for a parallel worker there might already be data on the current line
+ * (cf. ExplainOpenWorker); in that case, don't indent any more.
+ */
+static void
+ExplainIndentText(ExplainState *es)
+{
+ Assert(es->format == EXPLAIN_FORMAT_TEXT);
+ if (es->str->len == 0 || es->str->data[es->str->len - 1] == '\n')
+ appendStringInfoSpaces(es->str, es->indent * 2);
+}
+
+/*
* Emit a JSON line ending.
*
* JSON requires a comma after each property but the last. To facilitate this,
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7..853e206 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -25,6 +25,15 @@ typedef enum ExplainFormat
EXPLAIN_FORMAT_YAML
} ExplainFormat;
+typedef struct ExplainWorkersState
+{
+ int num_workers; /* # of worker processes the plan used */
+ bool *worker_inited; /* per-worker state-initialized flags */
+ StringInfoData *worker_str; /* per-worker transient output buffers */
+ int *worker_state_save; /* per-worker grouping state save areas */
+ StringInfo prev_str; /* saved output buffer while redirecting */
+} ExplainWorkersState;
+
typedef struct ExplainState
{
StringInfo str; /* output buffer */
@@ -47,6 +56,8 @@ typedef struct ExplainState
List *deparse_cxt; /* context list for deparsing expressions */
Bitmapset *printed_subplans; /* ids of SubPlans we've printed */
bool hide_workers; /* set if we find an invisible Gather */
+ /* state related to the current plan node */
+ ExplainWorkersState *workers_state; /* needed if parallel plan */
} ExplainState;
/* Hook for plugins to get control in ExplainOneQuery() */
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-25 03:33 Maciek Sakrejda <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Maciek Sakrejda @ 2020-01-25 03:33 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Georgios Kokolatos <[email protected]>; [email protected]
Thanks for the thorough review. I obviously missed some critical
issues. I recognized some of the other maintainability problems, but
did not have a sense of how to best avoid them, being unfamiliar with
the code.
For what it's worth, this version makes sense to me.
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-25 19:23 Tom Lane <[email protected]>
parent: Maciek Sakrejda <[email protected]>
0 siblings, 2 replies; 288+ messages in thread
From: Tom Lane @ 2020-01-25 19:23 UTC (permalink / raw)
To: Maciek Sakrejda <[email protected]>; +Cc: Andres Freund <[email protected]>; Georgios Kokolatos <[email protected]>; [email protected]
Maciek Sakrejda <[email protected]> writes:
> For what it's worth, this version makes sense to me.
Thanks for looking. Here's a version that deals with the JIT
instrumentation. As Andres noted far upthread, that was also
really bogusly done before. Not only could you get multiple "JIT"
subnodes on a Gather node, but we failed to print the info at all
if the parallelism was expressed as Gather Merge rather than
Gather.
A side effect of this change is that per-worker JIT info is now
printed one plan level further down: before it was printed on
the Gather node, but now it's attached to the Gather's child,
because that's where we print other per-worker data. I don't
see anything particularly wrong with that, but it's another
change from the behavior today.
It's still really unclear to me how we could exercise any of
this behavior meaningfully in a regression test. I thought
for a little bit about using the TAP infrastructure instead
of a traditional-style test, but it seems like that doesn't
buy anything except for a bias towards ignoring details instead
of overspecifying them. Which is not much of an improvement.
regards, tom lane
Attachments:
[text/x-diff] merge-explain-worker-output-v7.patch (27.5K, ../../[email protected]/2-merge-explain-worker-output-v7.patch)
download | inline diff:
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d189b8d..fe3c83b 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -57,6 +57,8 @@ static void ExplainOneQuery(Query *query, int cursorOptions,
IntoClause *into, ExplainState *es,
const char *queryString, ParamListInfo params,
QueryEnvironment *queryEnv);
+static void ExplainPrintJIT(ExplainState *es, int jit_flags,
+ JitInstrumentation *ji);
static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
ExplainState *es);
static double elapsed_time(instr_time *starttime);
@@ -123,11 +125,20 @@ static void ExplainSubPlans(List *plans, List *ancestors,
const char *relationship, ExplainState *es);
static void ExplainCustomChildren(CustomScanState *css,
List *ancestors, ExplainState *es);
+static ExplainWorkersState *ExplainCreateWorkersState(int num_workers);
+static void ExplainOpenWorker(int n, ExplainState *es);
+static void ExplainCloseWorker(int n, ExplainState *es);
+static void ExplainFlushWorkersState(ExplainState *es);
static void ExplainProperty(const char *qlabel, const char *unit,
const char *value, bool numeric, ExplainState *es);
+static void ExplainOpenSetAsideGroup(const char *objtype, const char *labelname,
+ bool labeled, int depth, ExplainState *es);
+static void ExplainSaveGroup(ExplainState *es, int depth, int *state_save);
+static void ExplainRestoreGroup(ExplainState *es, int depth, int *state_save);
static void ExplainDummyGroup(const char *objtype, const char *labelname,
ExplainState *es);
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
+static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -790,31 +801,22 @@ ExplainPrintJITSummary(ExplainState *es, QueryDesc *queryDesc)
if (queryDesc->estate->es_jit_worker_instr)
InstrJitAgg(&ji, queryDesc->estate->es_jit_worker_instr);
- ExplainPrintJIT(es, queryDesc->estate->es_jit_flags, &ji, -1);
+ ExplainPrintJIT(es, queryDesc->estate->es_jit_flags, &ji);
}
/*
* ExplainPrintJIT -
* Append information about JITing to es->str.
- *
- * Can be used to print the JIT instrumentation of the backend (worker_num =
- * -1) or that of a specific worker (worker_num = ...).
*/
-void
-ExplainPrintJIT(ExplainState *es, int jit_flags,
- JitInstrumentation *ji, int worker_num)
+static void
+ExplainPrintJIT(ExplainState *es, int jit_flags, JitInstrumentation *ji)
{
instr_time total_time;
- bool for_workers = (worker_num >= 0);
/* don't print information if no JITing happened */
if (!ji || ji->created_functions == 0)
return;
- /* don't print per-worker info if we're supposed to hide that */
- if (for_workers && es->hide_workers)
- return;
-
/* calculate total time */
INSTR_TIME_SET_ZERO(total_time);
INSTR_TIME_ADD(total_time, ji->generation_counter);
@@ -827,16 +829,13 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
/* for higher density, open code the text output format */
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (for_workers)
- appendStringInfo(es->str, "JIT for worker %u:\n", worker_num);
- else
- appendStringInfoString(es->str, "JIT:\n");
- es->indent += 1;
+ ExplainIndentText(es);
+ appendStringInfoString(es->str, "JIT:\n");
+ es->indent++;
ExplainPropertyInteger("Functions", NULL, ji->created_functions, es);
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Options: %s %s, %s %s, %s %s, %s %s\n",
"Inlining", jit_flags & PGJIT_INLINE ? "true" : "false",
"Optimization", jit_flags & PGJIT_OPT3 ? "true" : "false",
@@ -845,7 +844,7 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
if (es->analyze && es->timing)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Timing: %s %.3f ms, %s %.3f ms, %s %.3f ms, %s %.3f ms, %s %.3f ms\n",
"Generation", 1000.0 * INSTR_TIME_GET_DOUBLE(ji->generation_counter),
@@ -855,11 +854,10 @@ ExplainPrintJIT(ExplainState *es, int jit_flags,
"Total", 1000.0 * INSTR_TIME_GET_DOUBLE(total_time));
}
- es->indent -= 1;
+ es->indent--;
}
else
{
- ExplainPropertyInteger("Worker Number", NULL, worker_num, es);
ExplainPropertyInteger("Functions", NULL, ji->created_functions, es);
ExplainOpenGroup("Options", "Options", true, es);
@@ -1074,9 +1072,10 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
* optional name to be attached to the node.
*
* In text format, es->indent is controlled in this function since we only
- * want it to change at plan-node boundaries. In non-text formats, es->indent
- * corresponds to the nesting depth of logical output groups, and therefore
- * is controlled by ExplainOpenGroup/ExplainCloseGroup.
+ * want it to change at plan-node boundaries (but a few subroutines will
+ * transiently increment it). In non-text formats, es->indent corresponds
+ * to the nesting depth of logical output groups, and therefore is controlled
+ * by ExplainOpenGroup/ExplainCloseGroup.
*/
static void
ExplainNode(PlanState *planstate, List *ancestors,
@@ -1090,9 +1089,20 @@ ExplainNode(PlanState *planstate, List *ancestors,
const char *partialmode = NULL;
const char *operation = NULL;
const char *custom_name = NULL;
+ ExplainWorkersState *save_workers_state = es->workers_state;
int save_indent = es->indent;
bool haschildren;
+ /*
+ * Prepare per-worker output buffers, if needed. We'll append the data in
+ * these to the main output string further down.
+ */
+ if (planstate->worker_instrument && es->analyze && !es->hide_workers)
+ es->workers_state = ExplainCreateWorkersState(planstate->worker_instrument->num_workers);
+ else
+ es->workers_state = NULL;
+
+ /* Identify plan node type, and print generic details */
switch (nodeTag(plan))
{
case T_Result:
@@ -1324,13 +1334,13 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
if (plan_name)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "%s\n", plan_name);
es->indent++;
}
if (es->indent)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "-> ");
es->indent += 2;
}
@@ -1574,6 +1584,56 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (es->format == EXPLAIN_FORMAT_TEXT)
appendStringInfoChar(es->str, '\n');
+ /* prepare per-worker general execution details */
+ if (es->workers_state && es->verbose)
+ {
+ WorkerInstrumentation *w = planstate->worker_instrument;
+
+ for (int n = 0; n < w->num_workers; n++)
+ {
+ Instrumentation *instrument = &w->instrument[n];
+ double nloops = instrument->nloops;
+ double startup_ms;
+ double total_ms;
+ double rows;
+
+ if (nloops <= 0)
+ continue;
+ startup_ms = 1000.0 * instrument->startup / nloops;
+ total_ms = 1000.0 * instrument->total / nloops;
+ rows = instrument->ntuples / nloops;
+
+ ExplainOpenWorker(n, es);
+
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ ExplainIndentText(es);
+ if (es->timing)
+ appendStringInfo(es->str,
+ "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
+ startup_ms, total_ms, rows, nloops);
+ else
+ appendStringInfo(es->str,
+ "actual rows=%.0f loops=%.0f\n",
+ rows, nloops);
+ }
+ else
+ {
+ if (es->timing)
+ {
+ ExplainPropertyFloat("Actual Startup Time", "ms",
+ startup_ms, 3, es);
+ ExplainPropertyFloat("Actual Total Time", "ms",
+ total_ms, 3, es);
+ }
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
+ }
+
+ ExplainCloseWorker(n, es);
+ }
+ }
+
/* target list */
if (es->verbose)
show_plan_tlist(planstate, ancestors, es);
@@ -1684,24 +1744,6 @@ ExplainNode(PlanState *planstate, List *ancestors,
nworkers, es);
}
- /*
- * Print per-worker Jit instrumentation. Use same conditions
- * as for the leader's JIT instrumentation, see comment there.
- */
- if (es->costs && es->verbose &&
- outerPlanState(planstate)->worker_jit_instrument)
- {
- PlanState *child = outerPlanState(planstate);
- int n;
- SharedJitInstrumentation *w = child->worker_jit_instrument;
-
- for (n = 0; n < w->num_workers; ++n)
- {
- ExplainPrintJIT(es, child->state->es_jit_flags,
- &w->jit_instr[n], n);
- }
- }
-
if (gather->single_copy || es->format != EXPLAIN_FORMAT_TEXT)
ExplainPropertyBool("Single Copy", gather->single_copy, es);
}
@@ -1881,80 +1923,54 @@ ExplainNode(PlanState *planstate, List *ancestors,
break;
}
+ /*
+ * Prepare per-worker JIT instrumentation. As with the overall JIT
+ * summary, this is printed only if printing costs is enabled.
+ */
+ if (es->workers_state && es->costs && es->verbose)
+ {
+ SharedJitInstrumentation *w = planstate->worker_jit_instrument;
+
+ if (w)
+ {
+ for (int n = 0; n < w->num_workers; n++)
+ {
+ ExplainOpenWorker(n, es);
+ ExplainPrintJIT(es, planstate->state->es_jit_flags,
+ &w->jit_instr[n]);
+ ExplainCloseWorker(n, es);
+ }
+ }
+ }
+
/* Show buffer usage */
if (es->buffers && planstate->instrument)
show_buffer_usage(es, &planstate->instrument->bufusage);
- /* Show worker detail */
- if (es->analyze && es->verbose && !es->hide_workers &&
- planstate->worker_instrument)
+ /* Prepare per-worker buffer usage */
+ if (es->workers_state && es->buffers && es->verbose)
{
WorkerInstrumentation *w = planstate->worker_instrument;
- bool opened_group = false;
- int n;
- for (n = 0; n < w->num_workers; ++n)
+ for (int n = 0; n < w->num_workers; n++)
{
Instrumentation *instrument = &w->instrument[n];
double nloops = instrument->nloops;
- double startup_ms;
- double total_ms;
- double rows;
if (nloops <= 0)
continue;
- startup_ms = 1000.0 * instrument->startup / nloops;
- total_ms = 1000.0 * instrument->total / nloops;
- rows = instrument->ntuples / nloops;
-
- if (es->format == EXPLAIN_FORMAT_TEXT)
- {
- appendStringInfoSpaces(es->str, es->indent * 2);
- appendStringInfo(es->str, "Worker %d: ", n);
- if (es->timing)
- appendStringInfo(es->str,
- "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
- startup_ms, total_ms, rows, nloops);
- else
- appendStringInfo(es->str,
- "actual rows=%.0f loops=%.0f\n",
- rows, nloops);
- es->indent++;
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
- es->indent--;
- }
- else
- {
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
-
- if (es->timing)
- {
- ExplainPropertyFloat("Actual Startup Time", "ms",
- startup_ms, 3, es);
- ExplainPropertyFloat("Actual Total Time", "ms",
- total_ms, 3, es);
- }
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
- if (es->buffers)
- show_buffer_usage(es, &instrument->bufusage);
-
- ExplainCloseGroup("Worker", NULL, true, es);
- }
+ ExplainOpenWorker(n, es);
+ show_buffer_usage(es, &instrument->bufusage);
+ ExplainCloseWorker(n, es);
}
-
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
}
+ /* Show per-worker details for this plan node, then pop that stack */
+ if (es->workers_state)
+ ExplainFlushWorkersState(es);
+ es->workers_state = save_workers_state;
+
/* Get ready to display the child plans */
haschildren = planstate->initPlan ||
outerPlanState(planstate) ||
@@ -2525,7 +2541,7 @@ show_tablesample(TableSampleClause *tsc, PlanState *planstate,
{
bool first = true;
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Sampling: %s (", method_name);
foreach(lc, params)
{
@@ -2572,7 +2588,7 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
@@ -2588,12 +2604,14 @@ show_sort_info(SortState *sortstate, ExplainState *es)
* You might think we should just skip this stanza entirely when
* es->hide_workers is true, but then we'd get no sort-method output at
* all. We have to make it look like worker 0's data is top-level data.
- * Currently, we only bother with that for text-format output.
+ * This is easily done by just skipping the OpenWorker/CloseWorker calls.
+ * Currently, we don't worry about the possibility that there are multiple
+ * workers in such a case; if there are, duplicate output fields will be
+ * emitted.
*/
if (sortstate->shared_info != NULL)
{
int n;
- bool opened_group = false;
for (n = 0; n < sortstate->shared_info->num_workers; n++)
{
@@ -2609,32 +2627,26 @@ show_sort_info(SortState *sortstate, ExplainState *es)
spaceType = tuplesort_space_type_name(sinstrument->spaceType);
spaceUsed = sinstrument->spaceUsed;
+ if (es->workers_state)
+ ExplainOpenWorker(n, es);
+
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
- if (n > 0 || !es->hide_workers)
- appendStringInfo(es->str, "Worker %d: ", n);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Sort Method: %s %s: %ldkB\n",
sortMethod, spaceType, spaceUsed);
}
else
{
- if (!opened_group)
- {
- ExplainOpenGroup("Workers", "Workers", false, es);
- opened_group = true;
- }
- ExplainOpenGroup("Worker", NULL, true, es);
- ExplainPropertyInteger("Worker Number", NULL, n, es);
ExplainPropertyText("Sort Method", sortMethod, es);
ExplainPropertyInteger("Sort Space Used", "kB", spaceUsed, es);
ExplainPropertyText("Sort Space Type", spaceType, es);
- ExplainCloseGroup("Worker", NULL, true, es);
}
+
+ if (es->workers_state)
+ ExplainCloseWorker(n, es);
}
- if (opened_group)
- ExplainCloseGroup("Workers", "Workers", false, es);
}
}
@@ -2721,7 +2733,7 @@ show_hash_info(HashState *hashstate, ExplainState *es)
else if (hinstrument.nbatch_original != hinstrument.nbatch ||
hinstrument.nbuckets_original != hinstrument.nbuckets)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Buckets: %d (originally %d) Batches: %d (originally %d) Memory Usage: %ldkB\n",
hinstrument.nbuckets,
@@ -2732,7 +2744,7 @@ show_hash_info(HashState *hashstate, ExplainState *es)
}
else
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str,
"Buckets: %d Batches: %d Memory Usage: %ldkB\n",
hinstrument.nbuckets, hinstrument.nbatch,
@@ -2758,7 +2770,7 @@ show_tidbitmap_info(BitmapHeapScanState *planstate, ExplainState *es)
{
if (planstate->exact_pages > 0 || planstate->lossy_pages > 0)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "Heap Blocks:");
if (planstate->exact_pages > 0)
appendStringInfo(es->str, " exact=%ld", planstate->exact_pages);
@@ -2894,7 +2906,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage)
/* Show only positive counter values. */
if (has_shared || has_local || has_temp)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "Buffers:");
if (has_shared)
@@ -2949,7 +2961,7 @@ show_buffer_usage(ExplainState *es, const BufferUsage *usage)
/* As above, show only positive counter values. */
if (has_timing)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str, "I/O Timings:");
if (!INSTR_TIME_IS_ZERO(usage->blk_read_time))
appendStringInfo(es->str, " read=%0.3f",
@@ -3237,7 +3249,7 @@ show_modifytable_info(ModifyTableState *mtstate, List *ancestors,
*/
if (es->format == EXPLAIN_FORMAT_TEXT)
{
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfoString(es->str,
fdwroutine ? foperation : operation);
}
@@ -3427,6 +3439,158 @@ ExplainCustomChildren(CustomScanState *css, List *ancestors, ExplainState *es)
}
/*
+ * Create a per-plan-node workspace for collecting per-worker data.
+ *
+ * Output related to each worker will be temporarily "set aside" into a
+ * separate buffer, which we'll merge into the main output stream once
+ * we've processed all data for the plan node. This makes it feasible to
+ * generate a coherent sub-group of fields for each worker, even though the
+ * code that produces the fields is in several different places in this file.
+ * Formatting of such a set-aside field group is managed by
+ * ExplainOpenSetAsideGroup and ExplainSaveGroup/ExplainRestoreGroup.
+ */
+static ExplainWorkersState *
+ExplainCreateWorkersState(int num_workers)
+{
+ ExplainWorkersState *wstate;
+
+ wstate = (ExplainWorkersState *) palloc(sizeof(ExplainWorkersState));
+ wstate->num_workers = num_workers;
+ wstate->worker_inited = (bool *) palloc0(num_workers * sizeof(bool));
+ wstate->worker_str = (StringInfoData *)
+ palloc0(num_workers * sizeof(StringInfoData));
+ wstate->worker_state_save = (int *) palloc(num_workers * sizeof(int));
+ return wstate;
+}
+
+/*
+ * Begin or resume output into the set-aside group for worker N.
+ */
+static void
+ExplainOpenWorker(int n, ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ Assert(wstate);
+ Assert(n >= 0 && n < wstate->num_workers);
+
+ /* Save prior output buffer pointer */
+ wstate->prev_str = es->str;
+
+ if (!wstate->worker_inited[n])
+ {
+ /* First time through, so create the buffer for this worker */
+ initStringInfo(&wstate->worker_str[n]);
+ es->str = &wstate->worker_str[n];
+
+ /*
+ * Push suitable initial formatting state for this worker's field
+ * group. We allow one extra logical nesting level, since this group
+ * will eventually be wrapped in an outer "Workers" group.
+ */
+ ExplainOpenSetAsideGroup("Worker", NULL, true, 2, es);
+
+ /*
+ * In non-TEXT formats we always emit a "Worker Number" field, even if
+ * there's no other data for this worker.
+ */
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Worker Number", NULL, n, es);
+
+ wstate->worker_inited[n] = true;
+ }
+ else
+ {
+ /* Resuming output for a worker we've already emitted some data for */
+ es->str = &wstate->worker_str[n];
+
+ /* Restore formatting state saved by last ExplainCloseWorker() */
+ ExplainRestoreGroup(es, 2, &wstate->worker_state_save[n]);
+ }
+
+ /*
+ * In TEXT format, prefix the first output line for this worker with
+ * "Worker N:". Then, any additional lines should be indented one more
+ * stop than the "Worker N" line is.
+ */
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ if (es->str->len == 0)
+ {
+ ExplainIndentText(es);
+ appendStringInfo(es->str, "Worker %d: ", n);
+ }
+
+ es->indent++;
+ }
+}
+
+/*
+ * End output for worker N --- must pair with previous ExplainOpenWorker call
+ */
+static void
+ExplainCloseWorker(int n, ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ Assert(wstate);
+ Assert(n >= 0 && n < wstate->num_workers);
+ Assert(wstate->worker_inited[n]);
+
+ /*
+ * Save formatting state in case we do another ExplainOpenWorker(), then
+ * pop the formatting stack.
+ */
+ ExplainSaveGroup(es, 2, &wstate->worker_state_save[n]);
+
+ /*
+ * In TEXT format, if we didn't actually produce any output line(s) then
+ * truncate off the partial line. (This is to avoid bogus output if, for
+ * instance, show_buffer_usage chooses not to print anything for the
+ * worker.) Also fix up the indent level.
+ */
+ if (es->format == EXPLAIN_FORMAT_TEXT)
+ {
+ while (es->str->len > 0 && es->str->data[es->str->len - 1] != '\n')
+ es->str->data[--(es->str->len)] = '\0';
+
+ es->indent--;
+ }
+
+ /* Restore prior output buffer pointer */
+ es->str = wstate->prev_str;
+}
+
+/*
+ * Print per-worker info for current node, then free the ExplainWorkersState.
+ */
+static void
+ExplainFlushWorkersState(ExplainState *es)
+{
+ ExplainWorkersState *wstate = es->workers_state;
+
+ ExplainOpenGroup("Workers", "Workers", false, es);
+ for (int i = 0; i < wstate->num_workers; i++)
+ {
+ if (wstate->worker_inited[i])
+ {
+ /* This must match previous ExplainOpenSetAsideGroup call */
+ ExplainOpenGroup("Worker", NULL, true, es);
+ appendStringInfoString(es->str, wstate->worker_str[i].data);
+ ExplainCloseGroup("Worker", NULL, true, es);
+
+ pfree(wstate->worker_str[i].data);
+ }
+ }
+ ExplainCloseGroup("Workers", "Workers", false, es);
+
+ pfree(wstate->worker_inited);
+ pfree(wstate->worker_str);
+ pfree(wstate->worker_state_save);
+ pfree(wstate);
+}
+
+/*
* Explain a property, such as sort keys or targets, that takes the form of
* a list of unlabeled items. "data" is a list of C strings.
*/
@@ -3439,7 +3603,7 @@ ExplainPropertyList(const char *qlabel, List *data, ExplainState *es)
switch (es->format)
{
case EXPLAIN_FORMAT_TEXT:
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
appendStringInfo(es->str, "%s: ", qlabel);
foreach(lc, data)
{
@@ -3560,7 +3724,7 @@ ExplainProperty(const char *qlabel, const char *unit, const char *value,
switch (es->format)
{
case EXPLAIN_FORMAT_TEXT:
- appendStringInfoSpaces(es->str, es->indent * 2);
+ ExplainIndentText(es);
if (unit)
appendStringInfo(es->str, "%s: %s %s\n", qlabel, value, unit);
else
@@ -3752,6 +3916,117 @@ ExplainCloseGroup(const char *objtype, const char *labelname,
}
/*
+ * Open a group of related objects, without emitting actual data.
+ *
+ * Prepare the formatting state as though we were beginning a group with
+ * the identified properties, but don't actually emit anything. Output
+ * subsequent to this call can be redirected into a separate output buffer,
+ * and then eventually appended to the main output buffer after doing a
+ * regular ExplainOpenGroup call (with the same parameters).
+ *
+ * The extra "depth" parameter is the new group's depth compared to current.
+ * It could be more than one, in case the eventual output will be enclosed
+ * in additional nesting group levels. We assume we don't need to track
+ * formatting state for those levels while preparing this group's output.
+ *
+ * There is no ExplainCloseSetAsideGroup --- in current usage, we always
+ * pop this state with ExplainSaveGroup.
+ */
+static void
+ExplainOpenSetAsideGroup(const char *objtype, const char *labelname,
+ bool labeled, int depth, ExplainState *es)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->grouping_stack = lcons_int(0, es->grouping_stack);
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ if (labelname)
+ es->grouping_stack = lcons_int(1, es->grouping_stack);
+ else
+ es->grouping_stack = lcons_int(0, es->grouping_stack);
+ es->indent += depth;
+ break;
+ }
+}
+
+/*
+ * Pop one level of grouping state, allowing for a re-push later.
+ *
+ * This is typically used after ExplainOpenSetAsideGroup; pass the
+ * same "depth" used for that.
+ *
+ * This should not emit any output. If state needs to be saved,
+ * save it at *state_save. Currently, an integer save area is sufficient
+ * for all formats, but we might need to revisit that someday.
+ */
+static void
+ExplainSaveGroup(ExplainState *es, int depth, int *state_save)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent -= depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->indent -= depth;
+ *state_save = linitial_int(es->grouping_stack);
+ es->grouping_stack = list_delete_first(es->grouping_stack);
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ es->indent -= depth;
+ *state_save = linitial_int(es->grouping_stack);
+ es->grouping_stack = list_delete_first(es->grouping_stack);
+ break;
+ }
+}
+
+/*
+ * Re-push one level of grouping state, undoing the effects of ExplainSaveGroup.
+ */
+static void
+ExplainRestoreGroup(ExplainState *es, int depth, int *state_save)
+{
+ switch (es->format)
+ {
+ case EXPLAIN_FORMAT_TEXT:
+ /* nothing to do */
+ break;
+
+ case EXPLAIN_FORMAT_XML:
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_JSON:
+ es->grouping_stack = lcons_int(*state_save, es->grouping_stack);
+ es->indent += depth;
+ break;
+
+ case EXPLAIN_FORMAT_YAML:
+ es->grouping_stack = lcons_int(*state_save, es->grouping_stack);
+ es->indent += depth;
+ break;
+ }
+}
+
+/*
* Emit a "dummy" group that never has any members.
*
* objtype is the type of the group object, labelname is its label within
@@ -3913,6 +4188,21 @@ ExplainXMLTag(const char *tagname, int flags, ExplainState *es)
}
/*
+ * Indent a text-format line.
+ *
+ * We indent by two spaces per indentation level. However, when emitting
+ * data for a parallel worker there might already be data on the current line
+ * (cf. ExplainOpenWorker); in that case, don't indent any more.
+ */
+static void
+ExplainIndentText(ExplainState *es)
+{
+ Assert(es->format == EXPLAIN_FORMAT_TEXT);
+ if (es->str->len == 0 || es->str->data[es->str->len - 1] == '\n')
+ appendStringInfoSpaces(es->str, es->indent * 2);
+}
+
+/*
* Emit a JSON line ending.
*
* JSON requires a comma after each property but the last. To facilitate this,
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 45027a7..54f6240 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -25,6 +25,15 @@ typedef enum ExplainFormat
EXPLAIN_FORMAT_YAML
} ExplainFormat;
+typedef struct ExplainWorkersState
+{
+ int num_workers; /* # of worker processes the plan used */
+ bool *worker_inited; /* per-worker state-initialized flags */
+ StringInfoData *worker_str; /* per-worker transient output buffers */
+ int *worker_state_save; /* per-worker grouping state save areas */
+ StringInfo prev_str; /* saved output buffer while redirecting */
+} ExplainWorkersState;
+
typedef struct ExplainState
{
StringInfo str; /* output buffer */
@@ -47,6 +56,8 @@ typedef struct ExplainState
List *deparse_cxt; /* context list for deparsing expressions */
Bitmapset *printed_subplans; /* ids of SubPlans we've printed */
bool hide_workers; /* set if we find an invisible Gather */
+ /* state related to the current plan node */
+ ExplainWorkersState *workers_state; /* needed if parallel plan */
} ExplainState;
/* Hook for plugins to get control in ExplainOneQuery() */
@@ -84,8 +95,6 @@ extern void ExplainPrintPlan(ExplainState *es, QueryDesc *queryDesc);
extern void ExplainPrintTriggers(ExplainState *es, QueryDesc *queryDesc);
extern void ExplainPrintJITSummary(ExplainState *es, QueryDesc *queryDesc);
-extern void ExplainPrintJIT(ExplainState *es, int jit_flags,
- struct JitInstrumentation *jit_instr, int worker_num);
extern void ExplainQueryText(ExplainState *es, QueryDesc *queryDesc);
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-25 23:20 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 288+ messages in thread
From: Tom Lane @ 2020-01-25 23:20 UTC (permalink / raw)
To: Maciek Sakrejda <[email protected]>; +Cc: Andres Freund <[email protected]>; Georgios Kokolatos <[email protected]>; [email protected]
I wrote:
> It's still really unclear to me how we could exercise any of
> this behavior meaningfully in a regression test. I thought
> for a little bit about using the TAP infrastructure instead
> of a traditional-style test, but it seems like that doesn't
> buy anything except for a bias towards ignoring details instead
> of overspecifying them. Which is not much of an improvement.
After further thought, I decided that about the best we can do
is suppress the "Workers" field in the regression test's expected
output. This still gives us code coverage of the relevant code,
and we can check that the output is valid JSON before we strip it,
so it's not a dead loss.
I rewrote the test script a bit to add some coverage of XML and YAML
output formats, since we had exactly none before, and pushed it.
regards, tom lane
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-26 00:02 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 288+ messages in thread
From: Andres Freund @ 2020-01-26 00:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Maciek Sakrejda <[email protected]>; Georgios Kokolatos <[email protected]>; [email protected]
Hi,
On 2020-01-25 14:23:50 -0500, Tom Lane wrote:
> A side effect of this change is that per-worker JIT info is now
> printed one plan level further down: before it was printed on
> the Gather node, but now it's attached to the Gather's child,
> because that's where we print other per-worker data. I don't
> see anything particularly wrong with that, but it's another
> change from the behavior today.
Yea, I don't see any need to be bothered by that.
> It's still really unclear to me how we could exercise any of
> this behavior meaningfully in a regression test. I thought
> for a little bit about using the TAP infrastructure instead
> of a traditional-style test, but it seems like that doesn't
> buy anything except for a bias towards ignoring details instead
> of overspecifying them. Which is not much of an improvement.
Hm. I'd like to avoid needing TAP for this kind of thing, it'll just
make it much more expensive in just about all ways.
Testing JIT explain is "easy" enough I think, I've posted a patch in
another thread, which just skips over the region of the test if JIT is
not available. See e.g. 0008 in
https://www.postgresql.org/message-id/20191029000229.fkjmuld3g7f2jq7i%40alap3.anarazel.de
(that's a thread I'd love your input in btw)
It's harder for parallel query though. If parallel query were able to
reuse workers, we could "just" check at the beginning of the test if we
are able to get the workers we need, and then skip the rest of the tests
if not. But as things stand that doesn't guarantee anything.
I wonder if we could introduce a debug GUC that makes parallel worker
acquisition just retry in a loop, for a time determined by the GUC. That
obviously would be a bad idea to do in a production setup, but it could
be good enough for regression tests? There are some deadlock dangers,
but I'm not sure they really matter for the tests.
> + /* prepare per-worker general execution details */
> + if (es->workers_state && es->verbose)
> + {
> + WorkerInstrumentation *w = planstate->worker_instrument;
> +
> + for (int n = 0; n < w->num_workers; n++)
> + {
> + Instrumentation *instrument = &w->instrument[n];
> + double nloops = instrument->nloops;
> + double startup_ms;
> + double total_ms;
> + double rows;
> +
> + if (nloops <= 0)
> + continue;
> + startup_ms = 1000.0 * instrument->startup / nloops;
> + total_ms = 1000.0 * instrument->total / nloops;
> + rows = instrument->ntuples / nloops;
> +
> + ExplainOpenWorker(n, es);
> +
> + if (es->format == EXPLAIN_FORMAT_TEXT)
> + {
> + ExplainIndentText(es);
> + if (es->timing)
> + appendStringInfo(es->str,
> + "actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
> + startup_ms, total_ms, rows, nloops);
> + else
> + appendStringInfo(es->str,
> + "actual rows=%.0f loops=%.0f\n",
> + rows, nloops);
> + }
> + else
> + {
> + if (es->timing)
> + {
> + ExplainPropertyFloat("Actual Startup Time", "ms",
> + startup_ms, 3, es);
> + ExplainPropertyFloat("Actual Total Time", "ms",
> + total_ms, 3, es);
> + }
> + ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
> + ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
> + }
> +
> + ExplainCloseWorker(n, es);
> + }
> + }
I'd personally move this into a separate function, given the patches
moves the code around already. ExplainNode() is already hard enough to
navigate...
It probably also makes sense to move everything but the nloops <= 0,
ExplainOpenWorker/ExplainCloseWorker into its own function. As far as I
can tell it now should be identical between the non-parallel case?
> +/*
> + * Begin or resume output into the set-aside group for worker N.
> + */
> +static void
Would it make sense to make these functions non-static? It seems
plausible that code for a custom node or such would want to add
its own information?
> +ExplainOpenWorker(int n, ExplainState *es)
> +{
> + ExplainWorkersState *wstate = es->workers_state;
> +
> + Assert(wstate);
> + Assert(n >= 0 && n < wstate->num_workers);
> +
> + /* Save prior output buffer pointer */
> + wstate->prev_str = es->str;
> +
> + if (!wstate->worker_inited[n])
> + {
> + /* First time through, so create the buffer for this worker */
> + initStringInfo(&wstate->worker_str[n]);
> + es->str = &wstate->worker_str[n];
> +
> + /*
> + * Push suitable initial formatting state for this worker's field
> + * group. We allow one extra logical nesting level, since this group
> + * will eventually be wrapped in an outer "Workers" group.
> + */
> + ExplainOpenSetAsideGroup("Worker", NULL, true, 2, es);
> +
> + /*
> + * In non-TEXT formats we always emit a "Worker Number" field, even if
> + * there's no other data for this worker.
> + */
> + if (es->format != EXPLAIN_FORMAT_TEXT)
> + ExplainPropertyInteger("Worker Number", NULL, n, es);
> +
> + wstate->worker_inited[n] = true;
> + }
> + else
> + {
> + /* Resuming output for a worker we've already emitted some data for */
> + es->str = &wstate->worker_str[n];
> +
> + /* Restore formatting state saved by last ExplainCloseWorker() */
> + ExplainRestoreGroup(es, 2, &wstate->worker_state_save[n]);
> + }
> +
> + /*
> + * In TEXT format, prefix the first output line for this worker with
> + * "Worker N:". Then, any additional lines should be indented one more
> + * stop than the "Worker N" line is.
> + */
> + if (es->format == EXPLAIN_FORMAT_TEXT)
> + {
> + if (es->str->len == 0)
> + {
> + ExplainIndentText(es);
> + appendStringInfo(es->str, "Worker %d: ", n);
> + }
> +
> + es->indent++;
> + }
> +}
I don't quite get the Worker %d bit. Why are we not outputting that in
the !worker_inited block?
> +/*
> + * Print per-worker info for current node, then free the ExplainWorkersState.
> + */
> +static void
> +ExplainFlushWorkersState(ExplainState *es)
> +{
> + ExplainWorkersState *wstate = es->workers_state;
> +
> + ExplainOpenGroup("Workers", "Workers", false, es);
> + for (int i = 0; i < wstate->num_workers; i++)
> + {
> + if (wstate->worker_inited[i])
> + {
> + /* This must match previous ExplainOpenSetAsideGroup call */
> + ExplainOpenGroup("Worker", NULL, true, es);
> + appendStringInfoString(es->str, wstate->worker_str[i].data);
Probably never matters, but given we do have the string length already,
we could use appendBinaryStringInfo().
> + ExplainCloseGroup("Worker", NULL, true, es);
Not related to this patch: I never got why we maintain a grouping stack
for some things, but don't do it for the group properties
themselves.
> /*
> + * Open a group of related objects, without emitting actual data.
> + *
> + * Prepare the formatting state as though we were beginning a group with
> + * the identified properties, but don't actually emit anything. Output
> + * subsequent to this call can be redirected into a separate output buffer,
> + * and then eventually appended to the main output buffer after doing a
> + * regular ExplainOpenGroup call (with the same parameters).
> + *
> + * The extra "depth" parameter is the new group's depth compared to current.
> + * It could be more than one, in case the eventual output will be enclosed
> + * in additional nesting group levels. We assume we don't need to track
> + * formatting state for those levels while preparing this group's output.
> + *
> + * There is no ExplainCloseSetAsideGroup --- in current usage, we always
> + * pop this state with ExplainSaveGroup.
> + */
> +static void
> +ExplainOpenSetAsideGroup(const char *objtype, const char *labelname,
> + bool labeled, int depth, ExplainState *es)
> +{
> + switch (es->format)
> + {
> + case EXPLAIN_FORMAT_TEXT:
> + /* nothing to do */
> + break;
> +
> + case EXPLAIN_FORMAT_XML:
> + es->indent += depth;
> + break;
> +
> + case EXPLAIN_FORMAT_JSON:
> + es->grouping_stack = lcons_int(0, es->grouping_stack);
> + es->indent += depth;
> + break;
> +
> + case EXPLAIN_FORMAT_YAML:
> + if (labelname)
> + es->grouping_stack = lcons_int(1, es->grouping_stack);
> + else
> + es->grouping_stack = lcons_int(0, es->grouping_stack);
> + es->indent += depth;
> + break;
> + }
> +}
Hm. It might be worthwhile to rename ExplainOpenSetAsideGroup and use it
from ExplainOpenGroup()? Seems we could just call it after
ExplainOpenGroup()'s switch, and remove all the indent/grouping_stack
related code from ExplainOpenGroup().
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-26 00:30 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 288+ messages in thread
From: Tom Lane @ 2020-01-26 00:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Maciek Sakrejda <[email protected]>; Georgios Kokolatos <[email protected]>; [email protected]
Andres Freund <[email protected]> writes:
> I wonder if we could introduce a debug GUC that makes parallel worker
> acquisition just retry in a loop, for a time determined by the GUC. That
> obviously would be a bad idea to do in a production setup, but it could
> be good enough for regression tests? There are some deadlock dangers,
> but I'm not sure they really matter for the tests.
Hmmm .... might work. Seems like a better idea than "run it by itself"
as we have to do now.
> I'd personally move this into a separate function, given the patches
> moves the code around already. ExplainNode() is already hard enough to
> navigate...
Well, it was already inline in ExplainNode, so this just moved the
code a few lines. I'm not that excited about moving little bits of
that function out-of-line.
>> +/*
>> + * Begin or resume output into the set-aside group for worker N.
>> + */
>> +static void
> Would it make sense to make these functions non-static? It seems
> plausible that code for a custom node or such would want to add
> its own information?
Maybe, but seems to me that there'd be a whole lot of other infrastructure
needed to track additional per-worker state. I'd just as soon not
expose this stuff until (a) there's a concrete not hypothetical use case
and (b) it's been around long enough to feel comfortable that there's
nothing wrong with the design.
>> + /*
>> + * In TEXT format, prefix the first output line for this worker with
>> + * "Worker N:". Then, any additional lines should be indented one more
>> + * stop than the "Worker N" line is.
>> + */
> I don't quite get the Worker %d bit. Why are we not outputting that in
> the !worker_inited block?
We might strip it off again in ExplainCloseWorker, and then potentially
add it back again in a later ExplainOpenWorker. That could only happen
if an earlier ExplainOpen/CloseWorker fragment decides not to emit any
text and then a later one wants to do so. Which I'm pretty sure is
unreachable right now, but I don't think this code should assume that.
>> + appendStringInfoString(es->str, wstate->worker_str[i].data);
> Probably never matters, but given we do have the string length already,
> we could use appendBinaryStringInfo().
Ah, I was thinking about making that change but then forgot.
>> + ExplainCloseGroup("Worker", NULL, true, es);
> Not related to this patch: I never got why we maintain a grouping stack
> for some things, but don't do it for the group properties
> themselves.
Right now it'd just be extra overhead. If we ever have a case where it's
not convenient for an ExplainCloseGroup caller to provide the same data
as for ExplainOpenGroup, then I'd be on board with storing that info.
> Hm. It might be worthwhile to rename ExplainOpenSetAsideGroup and use it
> from ExplainOpenGroup()? Seems we could just call it after
> ExplainOpenGroup()'s switch, and remove all the indent/grouping_stack
> related code from ExplainOpenGroup().
Hmm. It seemed easier to me to keep them separate, but ...
I did consider a design in which, instead of ExplainOpenSetAsideGroup,
there was some function that would initialize the "state_save" area and
then you'd call the "restore" function to make that state active. It
seemed like that would be too dissimilar from ExplainOpenGroup --- but
conceivably, we could reimplement ExplainOpenGroup as calling the
initialize function and then the restore function (along with doing some
output). Not really sure that'd be an improvement though: it'd involve
less duplicate code, but the functions would individually be harder to
wrap your brain around.
regards, tom lane
^ permalink raw reply [nested|flat] 288+ messages in thread
* Re: Duplicate Workers entries in some EXPLAIN plans
@ 2020-01-26 23:00 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Tom Lane @ 2020-01-26 23:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Thomas Munro <[email protected]>; Maciek Sakrejda <[email protected]>; Georgios Kokolatos <[email protected]>; [email protected]
I wrote:
> Andres Freund <[email protected]> writes:
>> I wonder if we could introduce a debug GUC that makes parallel worker
>> acquisition just retry in a loop, for a time determined by the GUC. That
>> obviously would be a bad idea to do in a production setup, but it could
>> be good enough for regression tests? There are some deadlock dangers,
>> but I'm not sure they really matter for the tests.
> Hmmm .... might work. Seems like a better idea than "run it by itself"
> as we have to do now.
The more I think about this, the more it seems like a good idea, and
not only for regression test purposes. If you're about to launch a
query that will run for hours even with the max number of workers,
you don't want it to launch with less than that number just because
somebody else was eating a worker slot for a few milliseconds.
So I'm imagining a somewhat general-purpose GUC defined like
"max_delay_to_acquire_parallel_worker", measured say in milliseconds.
The default would be zero (current behavior: try once and give up),
but you could set it to small positive values if you have that kind
of production concern, while the regression tests could set it to big
positive values. This would alleviate all sorts of problems we have
with not being able to assume stable results from parallel worker
acquisition in the tests.
regards, tom lane
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
* [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86
@ 2025-07-26 00:57 Lukas Fittl <[email protected]>
0 siblings, 0 replies; 288+ messages in thread
From: Lukas Fittl @ 2025-07-26 00:57 UTC (permalink / raw)
We switch to using the time stamp counter (TSC) instead of clock_gettime()
to reduce overhead of EXPLAIN (ANALYZE, TIME ON). Tests showed that runtime
is reduced by around 10% for queries moving lots of rows through the plan.
For now this is only enabled on Linux/x86, in case the system clocksource is
reported as TSC. Relying on the Linux kernel simplifies the logic to detect
if the present TSC is usable (frequency invariant, synchronized between
sockets, etc.). In all other cases we fallback to clock_gettime().
Note, that we intentionally use RDTSC in the fast paths, rather than RDTSCP.
RDTSCP waits for outstanding instructions to retire on out-of-order CPUs.
This adds noticably for little benefit in the typical InstrStartNode() /
InstrStopNode() use case. The macro to be used in such cases is called
INSTR_TIME_SET_CURRENT_FAST(). The original macro INSTR_TIME_SET_CURRENT()
uses RDTSCP and is supposed to be used when precision is more important
than performance.
Author: David Geier <[email protected]>
Author: Andres Freund <[email protected]>
Author: Lukas Fittl <[email protected]>
Reviewed-by:
Discussion: https://www.postgresql.org/message-id/flat/20200612232810.f46nbqkdhbutzqdg%40alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 4 +-
src/backend/executor/instrument.c | 12 +-
src/backend/utils/init/postinit.c | 3 +
src/bin/pgbench/pgbench.c | 3 +
src/bin/psql/startup.c | 4 +
src/common/Makefile | 1 +
src/common/instr_time.c | 206 +++++++++++++++++++++++++++
src/common/meson.build | 1 +
src/include/portability/instr_time.h | 136 +++++++++++++++---
9 files changed, 348 insertions(+), 22 deletions(-)
create mode 100644 src/common/instr_time.c
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d2b031fdd06..5027048cac4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3409,8 +3409,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
INSTR_TIME_SET_CURRENT(currenttime);
elapsed = currenttime;
INSTR_TIME_SUBTRACT(elapsed, starttime);
- if ((INSTR_TIME_GET_MICROSEC(elapsed) / 1000)
- >= VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
+ if (INSTR_TIME_GET_MILLISEC(elapsed) >=
+ VACUUM_TRUNCATE_LOCK_CHECK_INTERVAL)
{
if (LockHasWaitersRelation(vacrel->rel, AccessExclusiveLock))
{
diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c
index 56e635f4700..01f67c5d972 100644
--- a/src/backend/executor/instrument.c
+++ b/src/backend/executor/instrument.c
@@ -67,9 +67,13 @@ InstrInit(Instrumentation *instr, int instrument_options)
void
InstrStartNode(Instrumentation *instr)
{
- if (instr->need_timer &&
- !INSTR_TIME_SET_CURRENT_LAZY(instr->starttime))
- elog(ERROR, "InstrStartNode called twice in a row");
+ if (instr->need_timer)
+ {
+ if (!INSTR_TIME_IS_ZERO(instr->starttime))
+ elog(ERROR, "InstrStartNode called twice in a row");
+ else
+ INSTR_TIME_SET_CURRENT_FAST(instr->starttime);
+ }
/* save buffer usage totals at node entry, if needed */
if (instr->need_bufusage)
@@ -95,7 +99,7 @@ InstrStopNode(Instrumentation *instr, double nTuples)
if (INSTR_TIME_IS_ZERO(instr->starttime))
elog(ERROR, "InstrStopNode called without start");
- INSTR_TIME_SET_CURRENT(endtime);
+ INSTR_TIME_SET_CURRENT_FAST(endtime);
INSTR_TIME_ACCUM_DIFF(instr->counter, endtime, instr->starttime);
INSTR_TIME_SET_ZERO(instr->starttime);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..d573409903b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -810,6 +810,9 @@ InitPostgres(const char *in_dbname, Oid dboid,
/* Initialize portal manager */
EnablePortalManager();
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/*
* Load relcache entries for the shared system catalogs. This must create
* at least entries for pg_database and catalogs used for authentication.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..79bef2d2aec 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7290,6 +7290,9 @@ main(int argc, char **argv)
initRandomState(&state[i].cs_func_rs);
}
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
/* opening connection... */
con = doConnect();
if (con == NULL)
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 249b6aa5169..d615df593c7 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -24,6 +24,7 @@
#include "help.h"
#include "input.h"
#include "mainloop.h"
+#include "portability/instr_time.h"
#include "settings.h"
/*
@@ -327,6 +328,9 @@ main(int argc, char *argv[])
PQsetNoticeProcessor(pset.db, NoticeProcessor, NULL);
+ /* initialize high-precision interval timing */
+ INSTR_TIME_INITIALIZE();
+
SyncVariables();
if (options.list_dbs)
diff --git a/src/common/Makefile b/src/common/Makefile
index 2c720caa509..1a2fbbe887f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -59,6 +59,7 @@ OBJS_COMMON = \
file_perm.o \
file_utils.o \
hashfn.o \
+ instr_time.o \
ip.o \
jsonapi.o \
keywords.o \
diff --git a/src/common/instr_time.c b/src/common/instr_time.c
new file mode 100644
index 00000000000..fdf47699f20
--- /dev/null
+++ b/src/common/instr_time.c
@@ -0,0 +1,206 @@
+/*-------------------------------------------------------------------------
+ *
+ * instr_time.c
+ * Non-inline parts of the portable high-precision interval timing
+ * implementation
+ *
+ * Portions Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/port/instr_time.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#if defined(HAVE__GET_CPUID) || (defined(HAVE__CPUIDEX) && !defined(_MSC_VER))
+#include <cpuid.h>
+#endif
+
+#if defined(HAVE__CPUID) || (defined(HAVE__CPUIDEX) && defined(_MSC_VER))
+#include <intrin.h>
+#endif
+
+#include "portability/instr_time.h"
+
+#ifndef WIN32
+/*
+ * Stores what the number of cycles needs to be multiplied with to end up
+ * with nanoseconds using integer math. See comment in pg_initialize_rdtsc()
+ * for more details.
+ *
+ * By default assume we are using clock_gettime() as a fallback which uses
+ * nanoseconds as ticks. Hence, we set the multiplier to the precision scalar
+ * so that the division in INSTR_TIME_GET_NANOSEC() won't change the nanoseconds.
+ *
+ * When using the RDTSC instruction directly this is filled in during initialization
+ * based on the relevant CPUID fields.
+ */
+int64 ticks_per_ns_scaled = TICKS_TO_NS_PRECISION;
+int64 ticks_per_sec = NS_PER_S;
+int64 max_ticks_no_overflow = PG_INT64_MAX / TICKS_TO_NS_PRECISION;
+
+#if defined(__x86_64__) && defined(__linux__)
+/*
+ * Indicates if RDTSC can be used (Linux/x86 only, when OS uses TSC clocksource)
+ */
+bool has_rdtsc = false;
+
+/*
+ * Indicates if RDTSCP can be used. True if RDTSC can be used and RDTSCP is available.
+ */
+bool has_rdtscp = false;
+
+#define CPUID_HYPERVISOR_VMWARE(words) (words[1] == 0x61774d56 && words[2] == 0x4d566572 && words[3] == 0x65726177) /* VMwareVMware */
+#define CPUID_HYPERVISOR_KVM(words) (words[1] == 0x4b4d564b && words[2] == 0x564b4d56 && words[3] == 0x0000004d) /* KVMKVMKVM */
+
+static bool
+get_tsc_frequency_khz(uint32 *tsc_freq)
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x15, &r[0] /* denominator */ , &r[1] /* numerator */ , &r[2] /* hz */ , &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x15);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[2] > 0)
+ {
+ if (r[0] == 0 || r[1] == 0)
+ return false;
+
+ *tsc_freq = r[2] / 1000 * r[1] / r[0];
+ return true;
+ }
+
+ /* Some CPUs only report frequency in 16H */
+
+#if defined(HAVE__GET_CPUID)
+ __get_cpuid(0x16, &r[0] /* base_mhz */ , &r[1], &r[2], &r[3]);
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x16);
+#else
+#error cpuid instruction not available
+#endif
+
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0] * 1000;
+ return true;
+ }
+
+ /*
+ * Check if we have a KVM or VMware Hypervisor passing down TSC frequency
+ * to us in a guest VM
+ *
+ * Note that accessing the 0x40000000 leaf for Hypervisor info requires
+ * use of __cpuidex to set ECX to 0. The similar __get_cpuid_count
+ * function does not work as expected since it contains a check for
+ * __get_cpuid_max, which has been observed to be lower than the special
+ * Hypervisor leaf.
+ */
+#if defined(HAVE__CPUIDEX)
+ __cpuidex((int32 *) r, 0x40000000, 0);
+ if (r[0] >= 0x40000010 && (CPUID_HYPERVISOR_VMWARE(r) || CPUID_HYPERVISOR_KVM(r)))
+ {
+ __cpuidex((int32 *) r, 0x40000010, 0);
+ if (r[0] > 0)
+ {
+ *tsc_freq = r[0];
+ return true;
+ }
+ }
+#endif
+
+ return false;
+}
+
+static bool
+is_rdtscp_available()
+{
+ uint32 r[4] = {0, 0, 0, 0};
+
+#if defined(HAVE__GET_CPUID)
+ if (!__get_cpuid(0x80000001, &r[0], &r[1], &r[2], &r[3]))
+ return false;
+#elif defined(HAVE__CPUID)
+ __cpuid(r, 0x80000001);
+#else
+#error cpuid instruction not available
+#endif
+
+ return (r[3] & (1 << 27)) != 0;
+}
+
+/*
+ * Decide whether we use the RDTSC instruction at runtime, for Linux/x86,
+ * instead of incurring the overhead of a full clock_gettime() call.
+ *
+ * This can't be reliably determined at compile time, since the
+ * availability of an "invariant" TSC (that is not affected by CPU
+ * frequency changes) is dependent on the CPU architecture. Additionally,
+ * there are cases where TSC availability is impacted by virtualization,
+ * where a simple cpuid feature check would not be enough.
+ *
+ * Since Linux already does a significant amount of work to determine
+ * whether TSC is a viable clock source, decide based on that.
+ */
+void
+pg_initialize_rdtsc(void)
+{
+ FILE *fp = fopen("/sys/devices/system/clocksource/clocksource0/current_clocksource", "r");
+
+ if (fp)
+ {
+ char buf[128];
+
+ if (fgets(buf, sizeof(buf), fp) != NULL && strcmp(buf, "tsc\n") == 0)
+ {
+ /*
+ * Compute baseline CPU peformance, determines speed at which
+ * RDTSC advances.
+ */
+ uint32 tsc_freq;
+
+ if (get_tsc_frequency_khz(&tsc_freq))
+ {
+ /*
+ * Ticks to nanoseconds conversion requires floating point
+ * math because because:
+ *
+ * sec = ticks / frequency_hz ns = ticks / frequency_hz *
+ * 1,000,000,000 ns = ticks * (1,000,000,000 / frequency_hz)
+ * ns = ticks * (1,000,000 / frequency_khz) <-- now in
+ * kilohertz
+ *
+ * Here, 'ns' is usually a floating number. For example for a
+ * 2.5 GHz CPU the scaling factor becomes 1,000,000 /
+ * 2,500,000 = 1.2.
+ *
+ * To be able to use integer math we work around the lack of
+ * precision. We first scale the integer up and after the
+ * multiplication by the number of ticks in
+ * INSTR_TIME_GET_NANOSEC() we divide again by the same value.
+ * We picked the scaler such that it provides enough precision
+ * and is a power-of-two which allows for shifting instead of
+ * doing an integer division.
+ */
+ ticks_per_ns_scaled = INT64CONST(1000000) * TICKS_TO_NS_PRECISION / tsc_freq;
+ ticks_per_sec = tsc_freq * 1000; /* KHz->Hz */
+ max_ticks_no_overflow = PG_INT64_MAX / ticks_per_ns_scaled;
+
+ has_rdtsc = true;
+ has_rdtscp = is_rdtscp_available();
+ }
+ }
+
+ fclose(fp);
+ }
+}
+#endif /* defined(__x86_64__) && defined(__linux__) */
+
+#endif /* WIN32 */
diff --git a/src/common/meson.build b/src/common/meson.build
index 1540ba67cca..62b90b3e609 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -13,6 +13,7 @@ common_sources = files(
'file_perm.c',
'file_utils.c',
'hashfn.c',
+ 'instr_time.c',
'ip.c',
'jsonapi.c',
'keywords.c',
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..e2e339a0c4f 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -4,9 +4,11 @@
* portable high-precision interval timing
*
* This file provides an abstraction layer to hide portability issues in
- * interval timing. On Unix we use clock_gettime(), and on Windows we use
- * QueryPerformanceCounter(). These macros also give some breathing room to
- * use other high-precision-timing APIs.
+ * interval timing. On Linux/x86 we use the rdtsc instruction when a TSC
+ * clocksource is also used on the host OS. Otherwise, and on other
+ * Unix-like systems we use clock_gettime() and on Windows we use
+ * QueryPerformanceCounter(). These macros also give some breathing
+ * room to use other high-precision-timing APIs.
*
* The basic data type is instr_time, which all callers should treat as an
* opaque typedef. instr_time can store either an absolute time (of
@@ -17,10 +19,11 @@
*
* INSTR_TIME_SET_ZERO(t) set t to zero (memset is acceptable too)
*
- * INSTR_TIME_SET_CURRENT(t) set t to current time
+ * INSTR_TIME_SET_CURRENT_FAST(t) set t to current time without waiting
+ * for instructions in out-of-order window
*
- * INSTR_TIME_SET_CURRENT_LAZY(t) set t to current time if t is zero,
- * evaluates to whether t changed
+ * INSTR_TIME_SET_CURRENT(t) set t to current time while waiting for
+ * instructions in OOO to retire
*
* INSTR_TIME_ADD(x, y) x += y
*
@@ -81,6 +84,15 @@ typedef struct instr_time
#ifndef WIN32
+/*
+ * Make sure this is a power-of-two, so that the compiler can turn the
+ * multiplications and divisions into shifts.
+ */
+#define TICKS_TO_NS_PRECISION (1<<14)
+
+extern int64 ticks_per_ns_scaled;
+extern int64 ticks_per_sec;
+extern int64 max_ticks_no_overflow;
/* Use clock_gettime() */
@@ -106,9 +118,18 @@ typedef struct instr_time
#define PG_INSTR_CLOCK CLOCK_REALTIME
#endif
-/* helper for INSTR_TIME_SET_CURRENT */
+#if defined(__x86_64__) && defined(__linux__)
+#include <x86intrin.h>
+#include <cpuid.h>
+
+extern bool has_rdtsc;
+extern bool has_rdtscp;
+
+extern void pg_initialize_rdtsc(void);
+#endif
+
static inline instr_time
-pg_clock_gettime_ns(void)
+pg_clock_gettime(void)
{
instr_time now;
struct timespec tmp;
@@ -119,11 +140,94 @@ pg_clock_gettime_ns(void)
return now;
}
+static inline instr_time
+pg_get_ticks_fast(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtsc)
+ {
+ instr_time now;
+
+ now.ticks = __rdtsc();
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline instr_time
+pg_get_ticks(void)
+{
+#if defined(__x86_64__) && defined(__linux__)
+ if (has_rdtscp)
+ {
+ instr_time now;
+ uint32 unused;
+
+ now.ticks = __rdtscp(&unused);
+ return now;
+ }
+#endif
+
+ return pg_clock_gettime();
+}
+
+static inline int64_t
+pg_ticks_to_ns(instr_time t)
+{
+ /*
+ * Would multiplication overflow? If so perform computation in two parts.
+ * Check overflow without actually overflowing via: a * b > max <=> a >
+ * max / b
+ */
+ int64 ns = 0;
+
+ if (unlikely(t.ticks > max_ticks_no_overflow))
+ {
+ /*
+ * Compute how often the maximum number of ticks fits completely into
+ * the number of elapsed ticks and convert that number into
+ * nanoseconds. Then multiply by the count to arrive at the final
+ * value. In a 2nd step we adjust the number of elapsed ticks and
+ * convert the remaining ticks.
+ */
+ int64 count = t.ticks / max_ticks_no_overflow;
+ int64 max_ns = max_ticks_no_overflow * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+
+ ns = max_ns * count;
+
+ /*
+ * Subtract the ticks that we now already accounted for, so that they
+ * don't get counted twice.
+ */
+ t.ticks -= count * max_ticks_no_overflow;
+ Assert(t.ticks >= 0);
+ }
+
+ ns += t.ticks * ticks_per_ns_scaled / TICKS_TO_NS_PRECISION;
+ return ns;
+}
+
+static inline void
+pg_initialize_get_ticks()
+{
+#if defined(__x86_64__) && defined(__linux__)
+ pg_initialize_rdtsc();
+#endif
+}
+
+#define INSTR_TIME_INITIALIZE() \
+ pg_initialize_get_ticks()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_get_ticks_fast())
+
#define INSTR_TIME_SET_CURRENT(t) \
- ((t) = pg_clock_gettime_ns())
+ ((t) = pg_get_ticks())
#define INSTR_TIME_GET_NANOSEC(t) \
- ((int64) (t).ticks)
+ pg_ticks_to_ns(t)
#else /* WIN32 */
@@ -131,7 +235,7 @@ pg_clock_gettime_ns(void)
/* Use QueryPerformanceCounter() */
-/* helper for INSTR_TIME_SET_CURRENT */
+/* helper for INSTR_TIME_SET_CURRENT / INSTR_TIME_SET_CURRENT_FAST */
static inline instr_time
pg_query_performance_counter(void)
{
@@ -153,6 +257,11 @@ GetTimerFrequency(void)
return (double) f.QuadPart;
}
+#define INSTR_TIME_INITIALIZE()
+
+#define INSTR_TIME_SET_CURRENT_FAST(t) \
+ ((t) = pg_query_performance_counter())
+
#define INSTR_TIME_SET_CURRENT(t) \
((t) = pg_query_performance_counter())
@@ -168,13 +277,8 @@ GetTimerFrequency(void)
#define INSTR_TIME_IS_ZERO(t) ((t).ticks == 0)
-
#define INSTR_TIME_SET_ZERO(t) ((t).ticks = 0)
-#define INSTR_TIME_SET_CURRENT_LAZY(t) \
- (INSTR_TIME_IS_ZERO(t) ? INSTR_TIME_SET_CURRENT(t), true : false)
-
-
#define INSTR_TIME_ADD(x,y) \
((x).ticks += (y).ticks)
--
2.47.3
--vtqqtrpooseurzip
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v12-0003-pg_test_timing-Add-fast-flag-to-test-fast-timing.patch"
^ permalink raw reply [nested|flat] 288+ messages in thread
end of thread, other threads:[~2025-07-26 00:57 UTC | newest]
Thread overview: 288+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-01-14 14:44 Re: Duplicate Workers entries in some EXPLAIN plans Georgios Kokolatos <[email protected]>
2020-01-15 07:22 ` Maciek Sakrejda <[email protected]>
2020-01-15 10:11 ` Georgios Kokolatos <[email protected]>
2020-01-15 17:12 ` Maciek Sakrejda <[email protected]>
2020-01-16 14:07 ` Georgios Kokolatos <[email protected]>
2020-01-21 08:48 ` Maciek Sakrejda <[email protected]>
2020-01-22 12:54 ` Georgios Kokolatos <[email protected]>
2020-01-22 16:54 ` Maciek Sakrejda <[email protected]>
2020-01-22 17:36 ` Georgios Kokolatos <[email protected]>
2020-01-23 09:00 ` Maciek Sakrejda <[email protected]>
2020-01-23 10:38 ` Georgios Kokolatos <[email protected]>
2020-01-24 17:03 ` Maciek Sakrejda <[email protected]>
2020-01-24 17:40 ` Tom Lane <[email protected]>
2020-01-25 00:26 ` Maciek Sakrejda <[email protected]>
2020-01-25 02:14 ` Tom Lane <[email protected]>
2020-01-25 03:33 ` Maciek Sakrejda <[email protected]>
2020-01-25 19:23 ` Tom Lane <[email protected]>
2020-01-25 23:20 ` Tom Lane <[email protected]>
2020-01-26 00:02 ` Andres Freund <[email protected]>
2020-01-26 00:30 ` Tom Lane <[email protected]>
2020-01-26 23:00 ` Tom Lane <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
2025-07-26 00:57 [PATCH v12 2/3] Use time stamp counter to measure time on Linux/x86 Lukas Fittl <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox