pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pg_walinspect: add functions to locate and list WAL by time and LSN
12+ messages / 3 participants
[nested] [flat]

* pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-09-18 04:05  Chao Li <li.evan.chao@gmail.com>
  0 siblings, 2 replies; 12+ messages in thread

From: Chao Li @ 2026-09-18 04:05 UTC (permalink / raw)
  To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi,

I'd like to propose adding two new SQL functions to pg_walinspect.

# Motivation

During HOW2026 in Jinan this April, a DBA talked to me that PG lacks an easy way to locate WAL files by time.

He described a scenario where somebody had done something wrong in a production database, for example accidentally deleting important data. The administrator knew that the incident had happened at around 1am the last night and wanted to confirm what had happened by analyzing WAL.

The problem was that, with thousands of WAL files in pg_wal, where should he start?

# Analysis

After looking into this use case, I think the underlying requirement is slightly different from simply finding a WAL file by timestamp.

Given an approximate point in time, the user really wants an LSN range that is likely to contain the WAL records associated with the operation of interest, for example a DELETE.

PostgreSQL already provides several related facilities:

* pg_ls_waldir() lists WAL files and their modification times.
* pg_split_walfile_name() extracts the segment number and timeline ID from a WAL file name.
* pg_walfile_name() maps an LSN to a WAL file name.
* pg_walinspect can inspect WAL records by LSN.
* pg_waldump can dump WAL records by WAL file or LSN range.
* There are also a number of third-party WAL-mining tools.

However, WAL file modification times don't necessarily reflect the chronological position of the WAL segment. For example, on my local cluster:
```
% ls -l
total 65536
-rw-------@ 1 chaol  staff  16777216 Sep 17 16:21 000000010000000000000001
-rw-------@ 1 chaol  staff  16777216 Sep 16 13:24 000000010000000000000002
drwx------@ 2 chaol  staff        64 Sep 16 13:17 archive_status
drwx------@ 2 chaol  staff        64 Sep 16 13:17 summaries
```

Here, segment 1 has a later modification time than segment 2. Since modification times do not necessarily follow WAL segment order, using them to locate the WAL corresponding to a particular point in time can be difficult and sometimes confusing.

So fundamentally, what is missing is an easy way to map a wall-clock time to a useful WAL/LSN range.

# Design

I propose adding two SQL functions.

## pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute’)

Given a target time and optional before and after intervals, it returns a WAL range around that time.

Not all WAL records contain timestamps, so the implementation reuses the existing timestamp-extraction logic from xlogrecovery.c, exposed by this patch as GetXLogRecordTimestamp(), to identify timestamp-bearing WAL records.

## pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL)

Given a start LSN and an optional end LSN, it returns the WAL files covering that LSN range.

Unlike pg_walfile_name(), which returns only a WAL file name for a single LSN, this function returns each relevant WAL file together with that segment's start and end LSNs.

# Implementation

Both functions are added to the pg_walinspect extension.

There is already quite a bit of reusable WAL-reading infrastructure there, and semantically it also seems like the natural place for this functionality.

The implementation builds on the existing pg_walinspect and XLog reader infrastructure. It does not introduce a new WAL format or a separate WAL parser.

# Demo

For example, suppose I deleted a tuple at around 16:11 yesterday. I can first locate the WAL range around that time:
```
evantest=# SELECT * FROM pg_get_wal_location_at_time('2026-09-17 16:11:08.317383+08', before=>interval '5 minute', after=>interval '5 minute');
       start_timestamp        | start_lsn  |        end_timestamp        |  end_lsn
------------------------------+------------+-----------------------------+------------
 2026-09-17 15:57:44.67121+08 | 0/01D05FA0 | 2026-09-18 10:56:13.9463+08 | 0/01D28900
(1 row)
```

Here, you may notice that end_lsn is much later than the specified timestamp plus 5 minutes. I chose this example intentionally.

As mentioned above, not all WAL records carry a timestamp. In this demo database, no timestamp-bearing WAL records were generated for quite some time after the target period, so the next usable timestamp did not appear until today. As a result, the returned WAL range is much wider than the requested five-minute window.

This also illustrates why the function returns an approximate WAL range based on timestamp-bearing records rather than an exact time-to-LSN mapping.

Then I can inspect that LSN range:
```
evantest=# SELECT start_lsn, end_lsn, xid, resource_manager, record_type FROM pg_get_wal_records_info('0/01D05FA0', '0/01D28900') WHERE record_type in ('DELETE', 'COMMIT');
 start_lsn  |  end_lsn   | xid | resource_manager | record_type
------------+------------+-----+------------------+-------------
 0/01D05FA0 | 0/01D05FC8 | 681 | Transaction      | COMMIT
 0/01D0B530 | 0/01D0B558 | 682 | Transaction      | COMMIT
 0/01D0B590 | 0/01D0B5C8 | 683 | Heap             | DELETE
 0/01D0B5C8 | 0/01D0B5F0 | 683 | Transaction      | COMMIT
(4 rows)
```

Now we know the LSN range for the DELETE. If we want to determine which WAL file contains it:
```
evantest=# SELECT * FROM pg_get_wal_files('0/01D0B590', '0/01D0B5C8');
         wal_file         | segment_start_lsn | segment_end_lsn
--------------------------+-------------------+-----------------
 000000010000000000000001 | 0/01000000        | 0/02000000
(1 row)
```

From there, the WAL file can be examined further with pg_waldump or other WAL-mining tools.

PFA v1. Reviews and comments and suggestions are greatly appreciated.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments:

v1-0001-pg_walinspect-add-functions-to-locate-and-list-WA.patch (application/octet-stream, 59.5K) download From e601df48e2fbd1b59c988fff33579c7cf5a171f9 Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Fri, 18 Sep 2026 11:51:04 +0800 Subject: [PATCH v1] pg_walinspect: add functions to locate and list WAL by time and LSN Add two functions to pg_walinspect for locating WAL around a wall-clock time and for listing the WAL segment files covering an LSN range. pg_get_wal_location_at_time() takes a target timestamp and optional before/after intervals and returns an LSN range bracketed by WAL records that contain timestamps. WAL segment modification times are used only as a starting-position hint; the returned boundaries are determined from timestamps stored in WAL records. Since most WAL records do not carry timestamps, the returned LSN range can be wider than the requested time range. pg_get_wal_files() returns the retained WAL segment files intersecting a specified LSN range, together with the complete start and end LSN of each segment. It also handles ranges crossing timeline switches in the current timeline's history. To avoid duplicating timestamp extraction logic, expose the existing getRecordTimestamp() helper from xlogrecovery.c as GetXLogRecordTimestamp() and reuse it from pg_walinspect. These functions are intended to make it easier to investigate database activity when the approximate time of an event is known but the corresponding WAL location is not. Author: Chao Li <lic@highgo.com> --- contrib/pg_walinspect/Makefile | 3 +- .../pg_walinspect/expected/oldextversions.out | 15 +- .../pg_walinspect/expected/pg_walinspect.out | 229 +++++++ contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 43 ++ contrib/pg_walinspect/pg_walinspect.c | 566 ++++++++++++++++++ contrib/pg_walinspect/pg_walinspect.control | 2 +- contrib/pg_walinspect/sql/oldextversions.sql | 5 +- contrib/pg_walinspect/sql/pg_walinspect.sql | 143 +++++ doc/src/sgml/pgwalinspect.sgml | 143 ++++- src/backend/access/transam/xlogrecovery.c | 17 +- src/include/access/xlogrecovery.h | 2 + 12 files changed, 1147 insertions(+), 22 deletions(-) create mode 100644 contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 22090f7716c..dc1c45ef7e9 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -7,7 +7,8 @@ OBJS = \ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log" EXTENSION = pg_walinspect -DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql +DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql \ + pg_walinspect--1.1--1.2.sql REGRESS = pg_walinspect oldextversions diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 89953af3d0e..6228ad9ef39 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -46,18 +46,21 @@ SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; --- List what version 1.1 contains. +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; +-- List what the current version contains. \dx+ pg_walinspect - Objects in extension "pg_walinspect" - Object description -------------------------------------------------------- + Objects in extension "pg_walinspect" + Object description +---------------------------------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) + function pg_get_wal_files(pg_lsn,pg_lsn) + function pg_get_wal_location_at_time(timestamp with time zone,interval,interval) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(4 rows) +(6 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index f955ff5d3c5..f5a1a0f9980 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -90,6 +90,185 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); t (1 row) +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + file_ok | start_ok | end_ok +---------+----------+-------- + t | t | t +(1 row) + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + one_file +---------- + t +(1 row) + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); + equal_lsn_ok +-------------- + t +(1 row) + +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); + default_end_ok +---------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +ERROR: WAL start LSN must be less than end LSN +SELECT * FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +ERROR: WAL start LSN must be less than end LSN +SELECT * FROM pg_get_wal_files('0/0', '0/1'); +ERROR: WAL segment needed for the requested range is missing +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== +-- Put a target timestamp between two WAL-logged transaction commits. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_target \gset +INSERT INTO sample_tbl VALUES (6, 6); +SELECT start_timestamp <= :'wal_time_target'::timestamptz AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '1 microsecond', + interval '1 microsecond'); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +SELECT start_timestamp <= :'wal_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +SELECT start_timestamp <= :'wal_time_target'::timestamptz AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz + interval '1 microsecond' + AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +SELECT start_timestamp <= :'wal_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz + interval '1 microsecond' + AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- COMMIT PREPARED records also provide commit-time anchors. +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT record_type = 'COMMIT_PREPARED' AS prepared_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + interval '1 microsecond', + interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + prepared_ok +------------- + t +(1 row) + +-- Non-transaction WAL records with timestamps can also be anchors. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +INSERT INTO sample_tbl VALUES (8, 8); +SELECT start_info.record_type = 'RESTORE_POINT' AS start_ok, + end_info.record_type = 'RESTORE_POINT' AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + interval '1 microsecond', + interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | end_ok +----------+-------- + t | t +(1 row) + +-- A future upper bound uses a timestamped WAL record at or before now. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'current_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + location.end_timestamp >= :'current_time_target'::timestamptz AND + location.end_timestamp <= clock_timestamp() AS capped_end_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + end_info.record_type = 'COMMIT' AS real_record_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + interval '1 microsecond', + interval '1 day') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | capped_end_ok | lsn_ok | real_record_ok +----------+---------------+--------+---------------- + t | t | t | t +(1 row) + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; + defaults_ok +------------- + t +(1 row) + +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '0', interval '1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '1 second', interval '0'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', interval '-1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + after => interval '-1 second'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 day 1 microsecond'); +ERROR: before interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + after => interval '1 day 1 microsecond'); +ERROR: after interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +ERROR: requested time precedes the available WAL range +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); +ERROR: requested time follows the available WAL range -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -183,6 +362,20 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -213,6 +406,20 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users. GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) @@ -223,6 +430,10 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -251,6 +462,20 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) @@ -259,6 +484,10 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 82ab186c340..15c77285e0d 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -20,6 +20,7 @@ install_data( 'pg_walinspect.control', 'pg_walinspect--1.0.sql', 'pg_walinspect--1.0--1.1.sql', + 'pg_walinspect--1.1--1.2.sql', kwargs: contrib_data_args, ) diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql new file mode 100644 index 00000000000..f72d99a3ceb --- /dev/null +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -0,0 +1,43 @@ +/* contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql */ + +-- complain if script is sourced in psql, rather than via ALTER EXTENSION +\echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit + +-- +-- pg_get_wal_location_at_time() +-- +CREATE FUNCTION pg_get_wal_location_at_time( + IN target_time timestamptz, + IN before interval DEFAULT '1 minute', + IN after interval DEFAULT '1 minute', + OUT start_timestamp timestamptz, + OUT start_lsn pg_lsn, + OUT end_timestamp timestamptz, + OUT end_lsn pg_lsn +) +AS 'MODULE_PATHNAME', 'pg_get_wal_location_at_time' +LANGUAGE C STRICT PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) + TO pg_read_server_files; + +-- +-- pg_get_wal_files() +-- +CREATE FUNCTION pg_get_wal_files( + IN start_lsn pg_lsn, + IN end_lsn pg_lsn DEFAULT NULL, + OUT wal_file text, + OUT segment_start_lsn pg_lsn, + OUT segment_end_lsn pg_lsn +) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'pg_get_wal_files' +LANGUAGE C PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO pg_read_server_files; diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index a172f9e2b40..2a6749abe9a 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -12,7 +12,10 @@ */ #include "postgres.h" +#include <sys/stat.h> + #include "access/htup_details.h" +#include "access/timeline.h" #include "access/xlog.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" @@ -22,9 +25,12 @@ #include "funcapi.h" #include "miscadmin.h" #include "port/pg_bitutils.h" +#include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgrprotos.h" #include "utils/pg_lsn.h" +#include "utils/timestamp.h" #include "utils/tuplestore.h" /* @@ -38,6 +44,8 @@ PG_MODULE_MAGIC_EXT( ); PG_FUNCTION_INFO_V1(pg_get_wal_block_info); +PG_FUNCTION_INFO_V1(pg_get_wal_files); +PG_FUNCTION_INFO_V1(pg_get_wal_location_at_time); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -68,6 +76,37 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Metadata for one retained segment in the current timeline's history. */ +typedef struct WalTimeSegment +{ + XLogSegNo segno; /* segment number, used for WAL ordering */ + TimeLineID tli; /* timeline containing this segment */ + TimestampTz mtime; /* search hint, never a record timestamp */ + bool scanned; /* segment has already been decoded */ +} WalTimeSegment; + +/* Selected WAL position and timestamp for one time boundary. */ +typedef struct WalTimeBoundary +{ + bool found; /* matching record was found */ + TimestampTz time; /* record's local event time */ + XLogRecPtr lsn; /* record's start LSN */ +} WalTimeBoundary; + +static int wal_time_segment_cmp(const void *a, const void *b); +static WalTimeSegment * GetWalTimeSegments(TimestampTz target_time, + int *nsegments, int *candidate); +static void ScanWalTimeSegment(WalTimeSegment * segment, + TimestampTz wanted_start, + TimestampTz wanted_end, + bool upper_bound_is_current, + WalTimeBoundary * lower, + WalTimeBoundary * upper); +static void ProbeLowerBoundary(WalTimeBoundary * boundary, + TimestampTz time, XLogRecPtr lsn); +static void ProbeUpperBoundary(WalTimeBoundary * boundary, + TimestampTz time, XLogRecPtr lsn); + /* * Return the LSN up to which the server has WAL. */ @@ -190,6 +229,222 @@ ReadNextXLogRecord(XLogReaderState *xlogreader) return record; } +/* qsort comparator that puts segments in WAL order. */ +static int +wal_time_segment_cmp(const void *a, const void *b) +{ + const WalTimeSegment *seg1 = (const WalTimeSegment *) a; + const WalTimeSegment *seg2 = (const WalTimeSegment *) b; + + if (seg1->segno < seg2->segno) + return -1; + if (seg1->segno > seg2->segno) + return 1; + return 0; +} + +/* + * Probe a timestamped record at or before the requested lower bound. Among + * records examined so far, prefer the latest time, breaking ties in favor of + * the later WAL record. + */ +static void +ProbeLowerBoundary(WalTimeBoundary * boundary, TimestampTz time, + XLogRecPtr lsn) +{ + if (!boundary->found || time > boundary->time || + (time == boundary->time && lsn > boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + } +} + +/* + * Probe a timestamped record at or after the requested upper bound. Among + * records examined so far, prefer the earliest time, breaking ties in favor + * of the earlier WAL record. + */ +static void +ProbeUpperBoundary(WalTimeBoundary * boundary, TimestampTz time, + XLogRecPtr lsn) +{ + if (!boundary->found || time < boundary->time || + (time == boundary->time && lsn < boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + } +} + +/* + * Get the WAL segments that form the history of the server's current + * timeline. Other timelines may have files with the same segment number in + * pg_wal, so choose the timeline that is valid at the end of each segment, + * as read_local_xlog_page_no_wait() does. Return the segments in WAL order + * and, if candidate is not NULL, set it to the segment whose modification + * time is closest to target_time. The modification time is only a + * starting-position hint. + */ +static WalTimeSegment * +GetWalTimeSegments(TimestampTz target_time, int *nsegments, int *candidate) +{ + WalTimeSegment *segments = NULL; + DIR *dir; + struct dirent *de; + List *history; + XLogRecPtr current_lsn; + TimeLineID current_tli; + int allocated = 0; + int count = 0; + uint64 best_distance = 0; + + if (!RecoveryInProgress()) + current_lsn = GetFlushRecPtr(&current_tli); + else + { + TimeLineID insert_tli; + + current_lsn = GetXLogReplayRecPtr(&current_tli); + insert_tli = GetWALInsertionTimeLineIfSet(); + if (insert_tli != 0) + current_tli = insert_tli; + } + + history = readTimeLineHistory(current_tli); + dir = AllocateDir(XLOGDIR); + while ((de = ReadDir(dir, XLOGDIR)) != NULL) + { + char path[MAXPGPATH]; + struct stat statbuf; + TimeLineID file_tli; + XLogSegNo segno; + XLogRecPtr seg_start; + XLogRecPtr seg_end; + + if (!IsXLogFileName(de->d_name)) + continue; + + XLogFromFileName(de->d_name, &file_tli, &segno, wal_segment_size); + seg_start = segno * wal_segment_size; + seg_end = seg_start + wal_segment_size - 1; + + if (seg_start >= current_lsn || + file_tli != tliOfPointInHistory(seg_end, history)) + continue; + + snprintf(path, sizeof(path), XLOGDIR "/%s", de->d_name); + if (stat(path, &statbuf) != 0) + ereport(ERROR, + errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", path)); + + if (count == allocated) + { + allocated = allocated ? allocated * 2 : 16; + if (segments == NULL) + segments = palloc_array(WalTimeSegment, allocated); + else + segments = repalloc_array(segments, WalTimeSegment, allocated); + } + + segments[count].segno = segno; + segments[count].tli = file_tli; + segments[count].mtime = time_t_to_timestamptz(statbuf.st_mtime); + segments[count].scanned = false; + count++; + } + FreeDir(dir); + list_free_deep(history); + + if (count == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no retained WAL segments are available")); + + qsort(segments, count, sizeof(WalTimeSegment), wal_time_segment_cmp); + + /* A segment number identifies at most one file in this timeline history. */ + for (int i = 1; i < count; i++) + { + if (segments[i - 1].segno == segments[i].segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("duplicate WAL segment in current timeline history")); + } + + if (candidate != NULL) + { + *candidate = 0; + for (int i = 0; i < count; i++) + { + uint64 distance; + + if (segments[i].mtime < target_time) + distance = TimestampDifferenceMicroseconds(segments[i].mtime, + target_time); + else + distance = TimestampDifferenceMicroseconds(target_time, + segments[i].mtime); + if (i == 0 || distance < best_distance) + { + best_distance = distance; + *candidate = i; + } + } + } + + *nsegments = count; + return segments; +} + +/* + * Inspect timestamped records beginning in one segment and update both + * requested boundaries. XLogReadRecord() may read the following segment to + * assemble a record crossing the boundary, but that record is attributed to + * the segment containing its starting LSN. + */ +static void +ScanWalTimeSegment(WalTimeSegment * segment, TimestampTz wanted_start, + TimestampTz wanted_end, bool upper_bound_is_current, + WalTimeBoundary * lower, + WalTimeBoundary * upper) +{ + XLogRecPtr seg_start = segment->segno * wal_segment_size; + XLogRecPtr seg_end = seg_start + wal_segment_size; + XLogReaderState *xlogreader; + + Assert(!segment->scanned); + segment->scanned = true; + xlogreader = InitXLogReaderState(seg_start); + + while (ReadNextXLogRecord(xlogreader)) + { + TimestampTz record_time; + + if (xlogreader->ReadRecPtr >= seg_end) + break; + + if (!GetXLogRecordTimestamp(xlogreader, &record_time)) + continue; + + if (record_time <= wanted_start) + ProbeLowerBoundary(lower, record_time, xlogreader->ReadRecPtr); + + if (upper_bound_is_current && record_time <= wanted_end) + ProbeLowerBoundary(upper, record_time, xlogreader->ReadRecPtr); + else if (!upper_bound_is_current && record_time >= wanted_end) + ProbeUpperBoundary(upper, record_time, xlogreader->ReadRecPtr); + + CHECK_FOR_INTERRUPTS(); + } + + pfree(xlogreader->private_data); + XLogReaderFree(xlogreader); +} + /* * Output values that make up a row describing caller's WAL record. * @@ -516,6 +771,317 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * Locate a WAL region around a wall-clock time range using timestamped WAL + * records. + * + * Segment modification times choose the initial segment only, and WAL order + * is used as a search heuristic. A future upper bound is capped at the + * current time and uses a timestamped WAL record at or before that time. On + * success, the boundaries form a forward WAL range. Record timestamps need + * not be monotonic with WAL position, so the anchors are not necessarily the + * globally closest timestamped records to the requested bounds. + */ +Datum +pg_get_wal_location_at_time(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_LOCATION_AT_TIME_COLS 4 + TimestampTz target_time = PG_GETARG_TIMESTAMPTZ(0); + Interval *before = PG_GETARG_INTERVAL_P(1); + Interval *after = PG_GETARG_INTERVAL_P(2); + Interval zero = {0}; + Interval one_day = {.day = 1}; + TimestampTz wanted_start; + TimestampTz wanted_end; + TimestampTz current_time; + bool upper_bound_is_current = false; + WalTimeSegment *segments; + WalTimeBoundary lower = {0}; + WalTimeBoundary upper = {0}; + Datum values[PG_GET_WAL_LOCATION_AT_TIME_COLS]; + bool nulls[PG_GET_WAL_LOCATION_AT_TIME_COLS] = {0}; + TupleDesc tupdesc; + HeapTuple tuple; + int nsegments; + int candidate; + int first_segment; + int last_segment; + int left; + int right; + int nscanned = 0; + char candidate_fname[MAXFNAMELEN]; + + if (TIMESTAMP_NOT_FINITE(target_time)) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("target time must be finite")); + if (INTERVAL_NOT_FINITE(before) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must not exceed one day")); + if (INTERVAL_NOT_FINITE(after) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must not exceed one day")); + + /* The timestamp functions provide the normal overflow checks. */ + wanted_start = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_mi_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(before))); + wanted_end = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_pl_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(after))); + current_time = GetCurrentTimestamp(); + + /* Cap a future upper bound at the time currently available. */ + if (wanted_end > current_time) + { + if (wanted_start >= current_time) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("The requested lower bound is not earlier than the current time.")); + + wanted_end = current_time; + upper_bound_is_current = true; + } + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + segments = GetWalTimeSegments(target_time, &nsegments, &candidate); + XLogFileName(candidate_fname, segments[candidate].tli, + segments[candidate].segno, wal_segment_size); + ereport(DEBUG1, + errmsg_internal("WAL time search selected segment %s from %d retained segments", + candidate_fname, + nsegments)); + + /* Find the contiguous retained WAL range containing the candidate. */ + first_segment = candidate; + while (first_segment > 0 && + segments[first_segment - 1].segno + 1 == + segments[first_segment].segno) + first_segment--; + last_segment = candidate; + while (last_segment + 1 < nsegments && + segments[last_segment].segno + 1 == + segments[last_segment + 1].segno) + last_segment++; + + ScanWalTimeSegment(&segments[candidate], wanted_start, wanted_end, + upper_bound_is_current, + &lower, &upper); + nscanned++; + left = candidate - 1; + right = candidate + 1; + + /* + * Use WAL order as a search heuristic, scanning outward only in the + * direction needed for each missing anchor. Do not cross a missing WAL + * segment. For a future upper bound, scan through the available WAL tail + * so that the end anchor is as recent as WAL order can establish. + */ + while ((!lower.found && left >= first_segment) || + ((upper_bound_is_current || !upper.found) && + right <= last_segment)) + { + if (!lower.found && left >= first_segment) + { + ScanWalTimeSegment(&segments[left--], wanted_start, wanted_end, + upper_bound_is_current, + &lower, &upper); + nscanned++; + } + if ((upper_bound_is_current || !upper.found) && + right <= last_segment) + { + ScanWalTimeSegment(&segments[right++], wanted_start, wanted_end, + upper_bound_is_current, + &lower, &upper); + nscanned++; + } + } + + /* + * Concurrent events or a clock discontinuity can put a needed timestamp + * in the unexpected direction, or can make the selected LSNs run + * backwards. In that case, scan the rest of this contiguous WAL range + * before reporting failure. + */ + if (!lower.found || !upper.found || lower.lsn > upper.lsn) + { + for (int i = first_segment; i <= last_segment; i++) + { + if (segments[i].scanned) + continue; + ScanWalTimeSegment(&segments[i], wanted_start, wanted_end, + upper_bound_is_current, + &lower, &upper); + nscanned++; + } + } + + ereport(DEBUG1, + errmsg_internal("WAL time search decoded %d of %d segments in the contiguous retained range", + nscanned, last_segment - first_segment + 1)); + + if (upper_bound_is_current && last_segment + 1 < nsegments) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) (segments[last_segment].segno + 1))); + + if (!lower.found) + { + if (first_segment > 0) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) (segments[first_segment].segno - 1))); + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time precedes the available WAL range"), + errdetail("No retained WAL record has a timestamp at or before the requested lower bound.")); + } + if (!upper.found) + { + if (last_segment + 1 < nsegments) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) (segments[last_segment].segno + 1))); + if (upper_bound_is_current) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("No retained WAL record has a timestamp at or before the current time.")); + else + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("No retained WAL record has a timestamp at or after the requested upper bound.")); + } + + if (lower.lsn >= upper.lsn) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("could not find a valid WAL range for the requested time window"), + errdetail("The matching time boundaries do not form a forward WAL range.")); + + values[0] = TimestampTzGetDatum(lower.time); + values[1] = LSNGetDatum(lower.lsn); + values[2] = TimestampTzGetDatum(upper.time); + values[3] = LSNGetDatum(upper.lsn); + + pfree(segments); + tuple = heap_form_tuple(tupdesc, values, nulls); + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +#undef PG_GET_WAL_LOCATION_AT_TIME_COLS +} + +/* + * List the retained WAL segment files intersecting [start_lsn, end_lsn), or + * the segment containing start_lsn when end_lsn is omitted or equal to it. + * Report complete segment boundaries, not the range clipped to the inputs. + */ +Datum +pg_get_wal_files(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_FILES_COLS 3 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + XLogSegNo first_segno; + XLogSegNo last_segno; + bool point_lookup; + WalTimeSegment *segments; + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + int nsegments; + int segment_index = 0; + + InitMaterializedSRF(fcinfo, 0); + + /* Preserve the empty-set behavior that STRICT provided for a NULL start. */ + if (PG_ARGISNULL(0)) + PG_RETURN_VOID(); + + start_lsn = PG_GETARG_LSN(0); + end_lsn = PG_ARGISNULL(1) ? start_lsn : PG_GETARG_LSN(1); + point_lookup = start_lsn == end_lsn; + + ValidateInputLSNs(start_lsn, &end_lsn); + if (!point_lookup && start_lsn >= end_lsn) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("WAL start LSN must be less than end LSN")); + + XLByteToSeg(start_lsn, first_segno, wal_segment_size); + if (point_lookup) + last_segno = first_segno; + else + XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); + segments = GetWalTimeSegments(0, &nsegments, NULL); + + for (XLogSegNo segno = first_segno;; segno++) + { + Datum values[PG_GET_WAL_FILES_COLS]; + bool nulls[PG_GET_WAL_FILES_COLS] = {0}; + XLogRecPtr segment_start_lsn = segno * wal_segment_size; + XLogRecPtr segment_end_lsn = segment_start_lsn + wal_segment_size; + char fname[MAXFNAMELEN]; + + while (segment_index < nsegments && + segments[segment_index].segno < segno) + segment_index++; + + if (segment_index >= nsegments || + segments[segment_index].segno != segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the requested range is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) segno)); + + XLogFileName(fname, segments[segment_index].tli, segno, + wal_segment_size); + values[0] = CStringGetTextDatum(fname); + values[1] = LSNGetDatum(segment_start_lsn); + values[2] = LSNGetDatum(segment_end_lsn); + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, + values, nulls); + + if (segno == last_segno) + break; + + CHECK_FOR_INTERRUPTS(); + } + + pfree(segments); + PG_RETURN_VOID(); +#undef PG_GET_WAL_FILES_COLS +} + /* * Validate start and end LSNs coming from the function inputs. * diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control index efa3cb2cfe5..5f574b865bb 100644 --- a/contrib/pg_walinspect/pg_walinspect.control +++ b/contrib/pg_walinspect/pg_walinspect.control @@ -1,5 +1,5 @@ # pg_walinspect extension comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log' -default_version = '1.1' +default_version = '1.2' module_pathname = '$libdir/pg_walinspect' relocatable = true diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql index e35c4f3ed15..b935a067b3e 100644 --- a/contrib/pg_walinspect/sql/oldextversions.sql +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -27,10 +27,11 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; --- List what version 1.1 contains. +-- List what the current version contains. \dx+ pg_walinspect SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 1e64a22d29a..481d60b01ce 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -51,6 +51,129 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +SELECT * FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +SELECT * FROM pg_get_wal_files('0/0', '0/1'); + +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== + +-- Put a target timestamp between two WAL-logged transaction commits. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_target \gset +INSERT INTO sample_tbl VALUES (6, 6); + +SELECT start_timestamp <= :'wal_time_target'::timestamptz AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '1 microsecond', + interval '1 microsecond'); + +SELECT start_timestamp <= :'wal_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + +SELECT start_timestamp <= :'wal_time_target'::timestamptz AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz + interval '1 microsecond' + AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + +SELECT start_timestamp <= :'wal_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_target'::timestamptz + interval '1 microsecond' + AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 microsecond', + after => interval '1 microsecond'); + +-- COMMIT PREPARED records also provide commit-time anchors. +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; + +SELECT record_type = 'COMMIT_PREPARED' AS prepared_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + interval '1 microsecond', + interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + +-- Non-transaction WAL records with timestamps can also be anchors. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +INSERT INTO sample_tbl VALUES (8, 8); + +SELECT start_info.record_type = 'RESTORE_POINT' AS start_ok, + end_info.record_type = 'RESTORE_POINT' AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + interval '1 microsecond', + interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +-- A future upper bound uses a timestamped WAL record at or before now. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'current_time_target'::timestamptz - interval '1 microsecond' + AS start_ok, + location.end_timestamp >= :'current_time_target'::timestamptz AND + location.end_timestamp <= clock_timestamp() AS capped_end_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + end_info.record_type = 'COMMIT' AS real_record_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + interval '1 microsecond', + interval '1 day') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '0', interval '1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + interval '1 second', interval '0'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + after => interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + before => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_target', + after => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -110,6 +233,10 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; @@ -122,6 +249,10 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE pg_read_server_files FROM regress_pg_walinspect; @@ -134,6 +265,10 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes @@ -143,6 +278,10 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; @@ -152,6 +291,10 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 79c3ead40bc..c8bfd2ccb8c 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -34,9 +34,8 @@ </note> <tip> <para> - All of the <filename>pg_walinspect</filename> functions that show - information about records that fall within a certain LSN range are - permissive about accepting <replaceable>end_lsn</replaceable> + All of the <filename>pg_walinspect</filename> functions that accept an LSN + range are permissive about accepting <replaceable>end_lsn</replaceable> arguments that are after the server's current LSN. Using an <replaceable>end_lsn</replaceable> <quote>from the future</quote> will not raise an error. @@ -60,6 +59,144 @@ <title>General Functions</title> <variablelist> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-files"> + <term> + <function>pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record</function> + </term> + + <listitem> + <para> + If <replaceable>end_lsn</replaceable> is specified and differs from + <replaceable>start_lsn</replaceable>, returns each retained WAL segment + file that intersects the half-open WAL range + [<replaceable>start_lsn</replaceable>, + <replaceable>end_lsn</replaceable>). If + <replaceable>end_lsn</replaceable> is omitted, is null, or equals + <replaceable>start_lsn</replaceable>, returns the segment containing + <replaceable>start_lsn</replaceable>. The + <structfield>wal_file</structfield> column contains the file name, while + <structfield>segment_start_lsn</structfield> and + <structfield>segment_end_lsn</structfield> contain the complete segment + boundaries rather than boundaries clipped to the requested range. + </para> + <para> + File names use the timeline valid for each segment in the current + timeline's history. This allows a returned range to cross a timeline + switch. The function reports an error if any required segment file is + not currently retained in <filename>pg_wal</filename>. + </para> + </listitem> + </varlistentry> + + <varlistentry id="pgwalinspect-funcs-pg-get-wal-location-at-time"> + <term> + <function>pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute') returns record</function> + </term> + + <listitem> + <para> + Locates a nearby WAL region using records containing local event + timestamps as historical time anchors. The requested time range extends + from <replaceable>target_time</replaceable> minus + <replaceable>before</replaceable> through + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable>. It returns + <structfield>start_timestamp</structfield>, + <structfield>start_lsn</structfield>, + <structfield>end_timestamp</structfield>, and + <structfield>end_lsn</structfield>. On successful return, the following + relationships hold: +<programlisting> +start_timestamp &lt;= target_time - before +start_lsn &lt; end_lsn +</programlisting> + If <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> is not later than the current time, this + additional relationship holds: +<programlisting> +end_timestamp &gt;= target_time + after +</programlisting> + Otherwise, the requested upper bound is capped at the current time, and + the function uses a timestamped WAL record at or before the current time + as the end boundary, so <literal>end_timestamp &lt;= current + time</literal>. It does not construct an end boundary from the current + WAL position. Thus, both returned boundaries come from actual + timestamped WAL records and form a valid forward WAL range. Both + interval arguments must be greater than zero and not greater than one + day. + </para> + <para> + Most WAL records do not contain a wall-clock timestamp. Historical + anchors can be transaction <literal>COMMIT</literal>, + <literal>COMMIT PREPARED</literal>, <literal>ABORT</literal>, + or <literal>ABORT PREPARED</literal> records, as well as restore point + records. These are the WAL record types from which PostgreSQL recovery + can extract timestamps. Their timestamps describe the corresponding + event; they do not assign timestamps to heap, index, or other records. + </para> + <para> + In particular, a commit timestamp describes when a transaction + committed, not when its earlier changes generated WAL. For example, a + transaction might update a row at 14:00 and commit at 15:00; the heap + WAL was generated around 14:00, while the commit record is timestamped + 15:00. The returned LSNs therefore do not identify the exact generation + time of heap, index, or other WAL records, nor do they guarantee that + every record generated during the requested wall-clock range lies + between the returned LSNs. On a cluster with little timestamped + activity, the returned timestamps can be far from the requested bounds. + </para> + <para> + WAL record timestamps are not guaranteed to increase with WAL position, + for example when concurrent transactions commit. The function avoids + decoding all retained WAL: it uses WAL segment modification times as a + starting-position hint, WAL order as a search heuristic, and nearby + timestamped records as time anchors. Therefore, the returned historical + records are not guaranteed to be the globally closest timestamped + records to either requested time bound. If suitable time boundaries + cannot form a forward WAL range, the function reports an error rather + than returning a range with the LSNs reversed. + </para> + <para> + Only WAL that is currently retained in <filename>pg_wal</filename> can + be searched. Unlike the other <application>pg_walinspect</application> + functions, this function may inspect retained WAL from the current + timeline's history as well as the current timeline. The function + reports an error when either boundary cannot be found or a missing WAL + segment prevents the search from reaching it. WAL segment modification + times are used internally to choose where to begin searching, but are + not used as record timestamps and are not part of the function's + semantics. + </para> + <para> + For example, a DBA can locate WAL around activity near a given time and + then inspect the returned range: +<programlisting> +SELECT * +FROM pg_get_wal_location_at_time( + '2026-08-01 14:35:00+08', + before =&gt; '10 minutes', + after =&gt; '10 minutes'); + +SELECT * +FROM pg_get_wal_files('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); + +SELECT * +FROM pg_get_wal_records_info('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); +</programlisting> + The corresponding retained WAL files can be obtained with + <function>pg_get_wal_files</function>. If the returned WAL range does + not contain the records of interest, increase + <replaceable>before</replaceable> if the relevant activity might have + occurred earlier, or increase <replaceable>after</replaceable> if it + might have occurred later, and repeat the search. Failure to find a + record in the first returned range does not necessarily mean that the + record is absent from retained WAL. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index fff8d57ac61..d1ebf111f99 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -356,7 +356,6 @@ static void xlog_outrec(StringInfo buf, XLogReaderState *record); static void xlog_block_info(StringInfo buf, XLogReaderState *record); static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, TimeLineID replayTLI); -static bool getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime); static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); @@ -2413,8 +2412,8 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, * Currently, only transaction commit/abort records and restore points contain * timestamps. */ -static bool -getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) +bool +GetXLogRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) { uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; uint8 xact_info = info & XLOG_XACT_OPMASK; @@ -2663,10 +2662,10 @@ recoveryStopsBefore(XLogReaderState *record) /* * Note: we must fetch recordXtime regardless of recoveryTarget setting. - * We don't expect getRecordTimestamp ever to fail, since we already know - * this is a commit or abort record; but test its result anyway. + * We don't expect GetXLogRecordTimestamp ever to fail, since we already + * know this is a commit or abort record; but test its result anyway. */ - if (getRecordTimestamp(record, &recordXtime) && + if (GetXLogRecordTimestamp(record, &recordXtime) && recoveryTarget == RECOVERY_TARGET_TIME) { /* @@ -2747,7 +2746,7 @@ recoveryStopsAfter(XLogReaderState *record) recoveryStopAfter = true; recoveryStopXid = InvalidTransactionId; recoveryStopLSN = InvalidXLogRecPtr; - (void) getRecordTimestamp(record, &recoveryStopTime); + (void) GetXLogRecordTimestamp(record, &recoveryStopTime); strlcpy(recoveryStopName, recordRestorePointData->rp_name, MAXFNAMELEN); ereport(LOG, @@ -2787,7 +2786,7 @@ recoveryStopsAfter(XLogReaderState *record) TransactionId recordXid; /* Update the last applied transaction timestamp */ - if (getRecordTimestamp(record, &recordXtime)) + if (GetXLogRecordTimestamp(record, &recordXtime)) SetLatestXTime(recordXtime); /* Extract the XID of the committed/aborted transaction */ @@ -3006,7 +3005,7 @@ recoveryApplyDelay(XLogReaderState *record) xact_info != XLOG_XACT_COMMIT_PREPARED) return false; - if (!getRecordTimestamp(record, &xtime)) + if (!GetXLogRecordTimestamp(record, &xtime)) return false; delayUntil = TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay); diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrecovery.h index 8786b6d3a8e..1f05050d7d7 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -223,6 +223,8 @@ extern void GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream); extern TimestampTz GetLatestXTime(void); extern TimestampTz GetCurrentChunkReplayStartTime(void); extern XLogRecPtr GetCurrentReplayRecPtr(TimeLineID *replayEndTLI); +extern bool GetXLogRecordTimestamp(XLogReaderState *record, + TimestampTz *recordXtime); extern bool PromoteIsTriggered(void); extern bool CheckPromoteSignal(void); -- 2.50.1 (Apple Git-155) =
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-09-29 22:31  surya poondla <suryapoondla4@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  1 sibling, 1 reply; 12+ messages in thread

From: surya poondla @ 2026-09-29 22:31 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi Chao,

Thanks for the patch. I think the use case is real. v1 applies cleanly to
current master (dca6a9e320e) and builds
without warnings; the pg_walinspect regression tests pass.

I have some comments below, starting with pg_get_wal_files(),

Regrading pg_get_wal_files():
1. A future end_lsn is rejected, which contradicts what this patch edits.

From the expected output:

  SELECT * FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF');
  ERROR:  WAL start LSN must be less than end LSN

The patch widens the doc tip to "All of the pg_walinspect functions that
accept an LSN range are permissive about accepting end_lsn arguments that
are after the server's current LSN", and the next paragraph says
FFFFFFFF/FFFFFFFF is equivalent to the current LSN. Neither statement holds
here.
The cause is that point_lookup is computed before ValidateInputLSNs() caps
end_lsn:

  ValidateInputLSNs(start_lsn, &end_lsn);
  point_lookup = (start_lsn == end_lsn);

With that order, the extra "start_lsn >= end_lsn" check can go. It also
adds a second copy of "WAL start LSN must be less than end LSN", the
message the other thread is changing to "less than or equal to", so
dropping it avoids two near-identical messages with different meanings.

2. pg_get_wal_files('0/0', '0/1') reports "WAL segment needed for the
requested range is missing" / "Segment 0 is not present in pg_wal".
Segment 0 never exists, so this sends the user looking for a retention
problem. The other functions reject the same input with "could not read WAL
at LSN 0/00000000", via the lsn < XLOG_BLCKSZ check in
InitXLogReaderState(). The same check would fit here.


Regarding the pg_get_wal_location_at_time():

1. If the upper bound is capped at the current time and no timestamped
record exists between target_time - before and now, both anchors resolve to
the
same record and the function fails with "could not find a valid WAL range
for the requested time window". On a quiet system, asking about something
that happened 30 seconds ago hits this. "Nothing timestamped since then"
isn't really an error. Returning the lower anchor with the current flush
LSN as the end seems more useful, even though the docs deliberately avoid
building an end boundary from the WAL position. What was the reasoning for
that?

2. Regarding the cost, some paths aren't bounded at all. If the lower
anchor isn't found, the left scan runs back to the start of the contiguous
retained range, and the rescue scan then decodes every segment that
remains. So any target older than the retained WAL, such as the
clock_timestamp() - interval '100 years' regression test, decodes all of
pg_wal only to report "requested time precedes the available WAL range". On
a cluster that keeps a lot of WAL, that's a lot of I/O to produce an
error. Could the rescue scan be limited, or skipped when the lower anchor
can't be found?

Similarly, the "WAL segment needed for the time search is missing" check
for a capped upper bound depends only on last_segment and nsegments, which
are known before any decoding. Moving it ahead of the scan avoids a long
scan that is certain to end in that error.

A related case: reading the code, if no retained record carries a timestamp
at all (a long-running bulk transaction that hasn't committed
yet, for example), the outward scan runs through the whole contiguous range
and the function then reports "requested time precedes the
available WAL range", even when target_time is well inside retained WAL. If
pg_wal also has a gap, it reports a missing segment instead, though
restoring that segment wouldn't help. This is also the most expensive case,
since the whole range has to be decoded to reach it, so a distinct
message such as "no retained WAL record contains a timestamp" would at
least tell the user what's actually wrong.

3. Regarding Timelines, GetWalTimeSegments() picks each segment's timeline
with tliOfPointInHistory(seg_end, history). Taking the timeline valid at the
segment's end looks right to me: after a promotion, the new timeline's copy
of the switch segment contains all WAL up to the switch point. Without
archiving, the old timeline's copy isn't renamed to .partial and stays in
pg_wal under the same segment number, so this filter is what keeps the two
apart. However, none of this is exercised. The regression tests run only on
timeline 1, and the module has no TAP suite. Given that both the commit
message and the docs claim ranges can cross a timeline switch, I think this
needs a TAP test that promotes a standby (with and without archiving).

4. The prepared_ok test and the two RESTORE_POINT checks assert that one
specific record is the chosen anchor. Any other timestamped record in the
same window, such as a commit from another session under installcheck or
from an autovacuum worker, changes the anchor and fails the test. Checking
that the record type is one of the timestamp-bearing types would be more
robust. Also, the four location queries after wal_time_target make the same
call. The fourth query's assertions imply the first three, and the first
differs only in passing the intervals positionally, so the second and third
can go.

Minor Nits:
1) GetXLogRecordTimestamp() only inspects the decoded record, and none of
its logic is specific to recovery. Would xlogreader.c be a better home for
it than xlogrecovery.c?
xlogreader.c is also built for frontend programs (pg_waldump compiles it
with -DFRONTEND), and the headers it would need, access/xact.h and
access/xlog_internal.h, are already included by the
rmgrdesc code that pg_waldump builds. Exporting it from xlogrecovery.c
makes it backend-only, so a later time-based filter in pg_waldump, which
today has
none, couldn't reuse it.
2) The "duplicate WAL segment in current timeline history" ereport looks
unreachable, because the tliOfPointInHistory() filter leaves at most one
file per segment number. It could be an Assert() or elog() rather than a
user-facing ERRCODE_DATA_EXCEPTION.
3) GetWalTimeSegments(0, &nsegments, NULL) in pg_get_wal_files() passes a
target_time that is never used. Moving the mtime-based candidate selection
into its own function would remove that argument.
4) GetWalTimeSegments() repeats GetCurrentLSN()'s flush/replay LSN logic
and adds the insert-timeline selection from read_local_xlog_page_guts().
Giving GetCurrentLSN() an optional TLI out-parameter would keep all three
in sync.
5) "Preserve the empty-set behavior that STRICT provided for a NULL start":
since this is a new function, there's no earlier behavior to preserve.

Finally, pg_get_wal_files() is useful on its own and much closer to ready.
Would you consider splitting the two functions into separate patches, so it
can move forward while the time-search semantics are worked out?

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-01 00:16  Chao Li <li.evan.chao@gmail.com>
  parent: surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Chao Li @ 2026-10-01 00:16 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>



> On Sep 30, 2026, at 06:31, surya poondla <suryapoondla4@gmail.com> wrote:
> 
> Hi Chao,
> 
> Thanks for the patch. I think the use case is real. v1 applies cleanly to current master (dca6a9e320e) and builds
> without warnings; the pg_walinspect regression tests pass.

Hi Surya,

Thank you so much for being the first reviewer of this patch.

> 
> I have some comments below, starting with pg_get_wal_files(),
> 
> Regrading pg_get_wal_files():
> 1. A future end_lsn is rejected, which contradicts what this patch edits.
> 
> From the expected output:
> 
>   SELECT * FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF');
>   ERROR:  WAL start LSN must be less than end LSN
> 
> The patch widens the doc tip to "All of the pg_walinspect functions that accept an LSN range are permissive about accepting end_lsn arguments that
> are after the server's current LSN", and the next paragraph says FFFFFFFF/FFFFFFFF is equivalent to the current LSN. Neither statement holds here. 
> The cause is that point_lookup is computed before ValidateInputLSNs() caps end_lsn:
> 
>   ValidateInputLSNs(start_lsn, &end_lsn);
>   point_lookup = (start_lsn == end_lsn);
> 
> With that order, the extra "start_lsn >= end_lsn" check can go. It also adds a second copy of "WAL start LSN must be less than end LSN", the
> message the other thread is changing to "less than or equal to", so dropping it avoids two near-identical messages with different meanings.
>  

Accepted.

> 2. pg_get_wal_files('0/0', '0/1') reports "WAL segment needed for the requested range is missing" / "Segment 0 is not present in pg_wal".
> Segment 0 never exists, so this sends the user looking for a retention problem. The other functions reject the same input with "could not read WAL
> at LSN 0/00000000", via the lsn < XLOG_BLCKSZ check in InitXLogReaderState(). The same check would fit here.
> 

Make sense.

> 
> Regarding the pg_get_wal_location_at_time():
> 
> 1. If the upper bound is capped at the current time and no timestamped record exists between target_time - before and now, both anchors resolve to the
> same record and the function fails with "could not find a valid WAL range for the requested time window". On a quiet system, asking about something
> that happened 30 seconds ago hits this. "Nothing timestamped since then" isn't really an error. Returning the lower anchor with the current flush
> LSN as the end seems more useful, even though the docs deliberately avoid building an end boundary from the WAL position. What was the reasoning for that?
> 
> 2. Regarding the cost, some paths aren't bounded at all. If the lower anchor isn't found, the left scan runs back to the start of the contiguous retained range, and the rescue scan then decodes every segment that remains. So any target older than the retained WAL, such as the clock_timestamp() - interval '100 years' regression test, decodes all of pg_wal only to report "requested time precedes the available WAL range". On a cluster that keeps a lot of WAL, that's a lot of I/O to produce an error. Could the rescue scan be limited, or skipped when the lower anchor can't be found?
> 
> Similarly, the "WAL segment needed for the time search is missing" check for a capped upper bound depends only on last_segment and nsegments, which
> are known before any decoding. Moving it ahead of the scan avoids a long scan that is certain to end in that error.
> 
> A related case: reading the code, if no retained record carries a timestamp at all (a long-running bulk transaction that hasn't committed
> yet, for example), the outward scan runs through the whole contiguous range and the function then reports "requested time precedes the
> available WAL range", even when target_time is well inside retained WAL. If pg_wal also has a gap, it reports a missing segment instead, though
> restoring that segment wouldn't help. This is also the most expensive case, since the whole range has to be decoded to reach it, so a distinct
> message such as "no retained WAL record contains a timestamp" would at least tell the user what's actually wrong.

I thought more about the design and made some significant changes in v2.

First, I switched to binary search for the lower and upper boundaries. I initially avoided this because timestamps of timestamped WAL records are not guaranteed to be ordered monotonically by WAL position. For various reasons, a record with an earlier timestamp can appear later in WAL, so binary search may return an imprecise result. However, considering that (1) large timestamp reordering should be uncommon on production clusters, and (2) the user-specified time range will most likely be an estimate, I think we can tolerate limited timestamp reordering in exchange for decoding significantly less WAL. The search also examines one additional segment in each direction to accommodate limited reordering. If the returned range does not contain the expected records, the user can widen the time range and repeat the search.

Second, when the search finds only one timestamped WAL record within the specified time range, the function no longer fails. Instead, it returns that record’s LSN as both start_lsn and end_lsn.

Third, to simplify the boundary semantics and implementation, the returned boundary timestamps are now within the specified time range. On successful return, start_timestamp is at or after the lower bound, while end_timestamp is at or before the upper bound. Because the search is approximate, the returned records are not guaranteed to be the globally closest timestamped records to those bounds. For example, suppose the requested upper bound is 10:00, and the timestamped records are:

WAL segment 10: 09:55
WAL segment 11: 10:05
WAL segment 12: 10:06
WAL segment 13: 09:59

The binary search may select segment 10 and examine the adjacent segment 11. It would return 09:55 as the upper boundary. However, the globally closest record at or before 10:00 is the 09:59 record in segment 13, which is missed because its timestamp is reordered by more than one segment. As mentioned earlier, this degree of timestamp reordering should be very uncommon on a production cluster. If the user expects the relevant record to be in segment 13, they can widen the upper boundary, for example to 10:10, and repeat the search.

> 3. Regarding Timelines, GetWalTimeSegments() picks each segment's timeline with tliOfPointInHistory(seg_end, history). Taking the timeline valid at the
> segment's end looks right to me: after a promotion, the new timeline's copy of the switch segment contains all WAL up to the switch point. Without
> archiving, the old timeline's copy isn't renamed to .partial and stays in pg_wal under the same segment number, so this filter is what keeps the two
> apart. However, none of this is exercised. The regression tests run only on timeline 1, and the module has no TAP suite. Given that both the commit
> message and the docs claim ranges can cross a timeline switch, I think this needs a TAP test that promotes a standby (with and without archiving).
> 

I was straggling whether or not to add a TAP test and ended up not adding one because the module didn’t have other TAP tests. I can add a TAP test for this timeline switch case.

> 4. The prepared_ok test and the two RESTORE_POINT checks assert that one specific record is the chosen anchor. Any other timestamped record in the
> same window, such as a commit from another session under installcheck or from an autovacuum worker, changes the anchor and fails the test. Checking
> that the record type is one of the timestamp-bearing types would be more robust.

Agreed.

> Also, the four location queries after wal_time_target make the same
> call. The fourth query's assertions imply the first three, and the first differs only in passing the intervals positionally, so the second and third can go.
> 

Yep, a bit redundant. Removed the fourth.

> Minor Nits:
> 1) GetXLogRecordTimestamp() only inspects the decoded record, and none of its logic is specific to recovery. Would xlogreader.c be a better home for it than xlogrecovery.c? 
> xlogreader.c is also built for frontend programs (pg_waldump compiles it with -DFRONTEND), and the headers it would need, access/xact.h and access/xlog_internal.h, are already included by the
> rmgrdesc code that pg_waldump builds. Exporting it from xlogrecovery.c makes it backend-only, so a later time-based filter in pg_waldump, which today has
> none, couldn't reuse it.

Good point. Accepted.

> 2) The "duplicate WAL segment in current timeline history" ereport looks unreachable, because the tliOfPointInHistory() filter leaves at most one
> file per segment number. It could be an Assert() or elog() rather than a user-facing ERRCODE_DATA_EXCEPTION.

Agreed.

> 3) GetWalTimeSegments(0, &nsegments, NULL) in pg_get_wal_files() passes a target_time that is never used. Moving the mtime-based candidate selection into its own function would remove that argument.

Agreed.

> 4) GetWalTimeSegments() repeats GetCurrentLSN()'s flush/replay LSN logic and adds the insert-timeline selection from read_local_xlog_page_guts(). Giving GetCurrentLSN() an optional TLI out-parameter would keep all three in sync.

Agreed.

> 5) "Preserve the empty-set behavior that STRICT provided for a NULL start": since this is a new function, there's no earlier behavior to preserve.

Agreed.

> 
> Finally, pg_get_wal_files() is useful on its own and much closer to ready. Would you consider splitting the two functions into separate patches, so it can move forward while the time-search semantics are worked out?

Good suggestion. I have split the two functions into two commits.

PFA v2.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments: download all (2 files, .tar.gz)

v2-0001-pg_walinspect-add-function-to-list-WAL-files-by-L.patch (application/octet-stream, 27.8K) download From 4538e2ea3d687f1cac5ddbccf4364b73c093e8da Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v2 1/2] pg_walinspect: add function to list WAL files by LSN Add pg_get_wal_files(), which returns the retained WAL segment files intersecting an LSN range, together with each segment's complete LSN boundaries. A point lookup can be requested by omitting end_lsn or making it equal to start_lsn. Choose segment file names from the server's current timeline history so that ranges spanning a timeline switch are handled correctly. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- contrib/pg_walinspect/Makefile | 4 +- .../pg_walinspect/expected/oldextversions.out | 8 +- .../pg_walinspect/expected/pg_walinspect.out | 73 ++++++ contrib/pg_walinspect/meson.build | 6 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 22 ++ contrib/pg_walinspect/pg_walinspect.c | 209 +++++++++++++++++- contrib/pg_walinspect/pg_walinspect.control | 2 +- contrib/pg_walinspect/sql/oldextversions.sql | 5 +- contrib/pg_walinspect/sql/pg_walinspect.sql | 34 +++ contrib/pg_walinspect/t/001_timeline.pl | 62 ++++++ doc/src/sgml/pgwalinspect.sgml | 34 ++- src/tools/pgindent/typedefs.list | 1 + 12 files changed, 441 insertions(+), 19 deletions(-) create mode 100644 contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql create mode 100644 contrib/pg_walinspect/t/001_timeline.pl diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 22090f7716c..b7651231077 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -7,9 +7,11 @@ OBJS = \ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log" EXTENSION = pg_walinspect -DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql +DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql \ + pg_walinspect--1.1--1.2.sql REGRESS = pg_walinspect oldextversions +TAP_TESTS = 1 REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_walinspect/walinspect.conf diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 89953af3d0e..311017d66ca 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -46,18 +46,20 @@ SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; --- List what version 1.1 contains. +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; +-- List what the current version contains. \dx+ pg_walinspect Objects in extension "pg_walinspect" Object description ------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) + function pg_get_wal_files(pg_lsn,pg_lsn) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(4 rows) +(5 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index f955ff5d3c5..34864c65ec7 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -90,6 +90,54 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); t (1 row) +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + file_ok | start_ok | end_ok +---------+----------+-------- + t | t | t +(1 row) + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + one_file +---------- + t +(1 row) + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); + equal_lsn_ok +-------------- + t +(1 row) + +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); + default_end_ok +---------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +ERROR: WAL start LSN must be less than end LSN +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); + future_end_ok +--------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files('0/0', '0/1'); +ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -183,6 +231,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -213,6 +268,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users. GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) @@ -223,6 +285,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -251,6 +315,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) @@ -259,6 +330,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 82ab186c340..6f8b4a6359a 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -20,6 +20,7 @@ install_data( 'pg_walinspect.control', 'pg_walinspect--1.0.sql', 'pg_walinspect--1.0--1.1.sql', + 'pg_walinspect--1.1--1.2.sql', kwargs: contrib_data_args, ) @@ -37,4 +38,9 @@ tests += { 'regress_args': ['--temp-config', files('walinspect.conf')], 'runningcheck': false, }, + 'tap': { + 'tests': [ + 't/001_timeline.pl', + ], + }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql new file mode 100644 index 00000000000..9cc9523b4dc --- /dev/null +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -0,0 +1,22 @@ +/* contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql */ + +-- complain if script is sourced in psql, rather than via ALTER EXTENSION +\echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit + +-- +-- pg_get_wal_files() +-- +CREATE FUNCTION pg_get_wal_files( + IN start_lsn pg_lsn, + IN end_lsn pg_lsn DEFAULT NULL, + OUT wal_file text, + OUT segment_start_lsn pg_lsn, + OUT segment_end_lsn pg_lsn +) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'pg_get_wal_files' +LANGUAGE C PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO pg_read_server_files; diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index a172f9e2b40..42f366bdb4e 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -13,6 +13,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/timeline.h" #include "access/xlog.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" @@ -22,6 +23,7 @@ #include "funcapi.h" #include "miscadmin.h" #include "port/pg_bitutils.h" +#include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" @@ -38,6 +40,7 @@ PG_MODULE_MAGIC_EXT( ); PG_FUNCTION_INFO_V1(pg_get_wal_block_info); +PG_FUNCTION_INFO_V1(pg_get_wal_files); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -45,7 +48,7 @@ PG_FUNCTION_INFO_V1(pg_get_wal_stats); PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal); static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn); -static XLogRecPtr GetCurrentLSN(void); +static XLogRecPtr GetCurrentLSN(TimeLineID *current_tli); static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn); static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader); static void GetWALRecordInfo(XLogReaderState *record, Datum *values, @@ -68,11 +71,21 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Metadata for one retained segment in the current timeline's history. */ +typedef struct WalTimeSegment +{ + XLogSegNo segno; /* segment number, used for WAL ordering */ + TimeLineID tli; /* timeline containing this segment */ +} WalTimeSegment; + +static int wal_time_segment_cmp(const void *a, const void *b); +static WalTimeSegment *GetWalTimeSegments(int *nsegments); + /* - * Return the LSN up to which the server has WAL. + * Return the LSN up to which the server has WAL, and optionally its timeline. */ static XLogRecPtr -GetCurrentLSN(void) +GetCurrentLSN(TimeLineID *current_tli) { XLogRecPtr curr_lsn; @@ -81,9 +94,15 @@ GetCurrentLSN(void) * callback read_local_xlog_page_no_wait does. */ if (!RecoveryInProgress()) - curr_lsn = GetFlushRecPtr(NULL); + curr_lsn = GetFlushRecPtr(current_tli); else - curr_lsn = GetXLogReplayRecPtr(NULL); + { + curr_lsn = GetXLogReplayRecPtr(current_tli); + if (current_tli != NULL && *current_tli == 0) + { + *current_tli = GetWALInsertionTimeLineIfSet(); + } + } Assert(XLogRecPtrIsValid(curr_lsn)); @@ -190,6 +209,94 @@ ReadNextXLogRecord(XLogReaderState *xlogreader) return record; } +/* qsort comparator that puts segments in WAL order. */ +static int +wal_time_segment_cmp(const void *a, const void *b) +{ + const WalTimeSegment *seg1 = (const WalTimeSegment *) a; + const WalTimeSegment *seg2 = (const WalTimeSegment *) b; + + if (seg1->segno < seg2->segno) + return -1; + if (seg1->segno > seg2->segno) + return 1; + return 0; +} + +/* + * Get the WAL segments that form the history of the server's current + * timeline. Other timelines may have files with the same segment number in + * pg_wal, so choose the timeline that is valid at the end of each segment, + * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. + */ +static WalTimeSegment * +GetWalTimeSegments(int *nsegments) +{ + WalTimeSegment *segments = NULL; + DIR *dir; + struct dirent *de; + List *history; + XLogRecPtr current_lsn; + TimeLineID current_tli; + int allocated = 0; + int count = 0; + + current_lsn = GetCurrentLSN(&current_tli); + + history = readTimeLineHistory(current_tli); + dir = AllocateDir(XLOGDIR); + while ((de = ReadDir(dir, XLOGDIR)) != NULL) + { + TimeLineID file_tli; + XLogSegNo segno; + XLogRecPtr seg_start; + XLogRecPtr seg_end; + + if (!IsXLogFileName(de->d_name)) + continue; + + XLogFromFileName(de->d_name, &file_tli, &segno, wal_segment_size); + seg_start = segno * wal_segment_size; + seg_end = seg_start + wal_segment_size - 1; + + if (seg_start >= current_lsn || + file_tli != tliOfPointInHistory(seg_end, history)) + continue; + + if (count == allocated) + { + allocated = allocated ? allocated * 2 : 16; + if (segments == NULL) + segments = palloc_array(WalTimeSegment, allocated); + else + segments = repalloc_array(segments, WalTimeSegment, allocated); + } + + MemSet(&segments[count], 0, sizeof(WalTimeSegment)); + segments[count].segno = segno; + segments[count].tli = file_tli; + count++; + } + FreeDir(dir); + list_free_deep(history); + + if (count == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no retained WAL segments are available")); + + qsort(segments, count, sizeof(WalTimeSegment), wal_time_segment_cmp); + +#ifdef USE_ASSERT_CHECKING + /* A segment number identifies at most one file in this timeline history. */ + for (int i = 1; i < count; i++) + Assert(segments[i - 1].segno != segments[i].segno); +#endif + + *nsegments = count; + return segments; +} + /* * Output values that make up a row describing caller's WAL record. * @@ -483,7 +590,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) HeapTuple tuple; lsn = PG_GETARG_LSN(0); - curr_lsn = GetCurrentLSN(); + curr_lsn = GetCurrentLSN(NULL); if (lsn > curr_lsn) ereport(ERROR, @@ -516,6 +623,90 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * List the retained WAL segment files intersecting [start_lsn, end_lsn), or + * the segment containing start_lsn when end_lsn is omitted or equal to it. + * Report complete segment boundaries, not the range clipped to the inputs. + */ +Datum +pg_get_wal_files(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_FILES_COLS 3 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + XLogSegNo first_segno; + XLogSegNo last_segno; + bool point_lookup; + WalTimeSegment *segments; + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + int nsegments; + int segment_index = 0; + + InitMaterializedSRF(fcinfo, 0); + + /* A NULL start produces an empty set. */ + if (PG_ARGISNULL(0)) + PG_RETURN_VOID(); + + start_lsn = PG_GETARG_LSN(0); + end_lsn = PG_ARGISNULL(1) ? start_lsn : PG_GETARG_LSN(1); + + if (start_lsn < XLOG_BLCKSZ) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not read WAL at LSN %X/%08X", + LSN_FORMAT_ARGS(start_lsn))); + + ValidateInputLSNs(start_lsn, &end_lsn); + point_lookup = start_lsn == end_lsn; + + + XLByteToSeg(start_lsn, first_segno, wal_segment_size); + if (point_lookup) + last_segno = first_segno; + else + XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); + segments = GetWalTimeSegments(&nsegments); + + for (XLogSegNo segno = first_segno;; segno++) + { + Datum values[PG_GET_WAL_FILES_COLS]; + bool nulls[PG_GET_WAL_FILES_COLS] = {0}; + XLogRecPtr segment_start_lsn = segno * wal_segment_size; + XLogRecPtr segment_end_lsn = segment_start_lsn + wal_segment_size; + char fname[MAXFNAMELEN]; + + while (segment_index < nsegments && + segments[segment_index].segno < segno) + segment_index++; + + if (segment_index >= nsegments || + segments[segment_index].segno != segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the requested range is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) segno)); + + XLogFileName(fname, segments[segment_index].tli, segno, + wal_segment_size); + values[0] = CStringGetTextDatum(fname); + values[1] = LSNGetDatum(segment_start_lsn); + values[2] = LSNGetDatum(segment_end_lsn); + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, + values, nulls); + + if (segno == last_segno) + break; + + CHECK_FOR_INTERRUPTS(); + } + + pfree(segments); + PG_RETURN_VOID(); +#undef PG_GET_WAL_FILES_COLS +} + /* * Validate start and end LSNs coming from the function inputs. * @@ -525,7 +716,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn) { - XLogRecPtr curr_lsn = GetCurrentLSN(); + XLogRecPtr curr_lsn = GetCurrentLSN(NULL); if (start_lsn > curr_lsn) ereport(ERROR, @@ -831,7 +1022,7 @@ Datum pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); if (start_lsn > end_lsn) ereport(ERROR, @@ -849,7 +1040,7 @@ Datum pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); bool stats_per_record = PG_GETARG_BOOL(1); if (start_lsn > end_lsn) diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control index efa3cb2cfe5..5f574b865bb 100644 --- a/contrib/pg_walinspect/pg_walinspect.control +++ b/contrib/pg_walinspect/pg_walinspect.control @@ -1,5 +1,5 @@ # pg_walinspect extension comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log' -default_version = '1.1' +default_version = '1.2' module_pathname = '$libdir/pg_walinspect' relocatable = true diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql index e35c4f3ed15..b935a067b3e 100644 --- a/contrib/pg_walinspect/sql/oldextversions.sql +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -27,10 +27,11 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; --- List what version 1.1 contains. +-- List what the current version contains. \dx+ pg_walinspect SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 1e64a22d29a..3176a17d855 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -51,6 +51,30 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +SELECT * FROM pg_get_wal_files('0/0', '0/1'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -110,6 +134,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; @@ -122,6 +148,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE pg_read_server_files FROM regress_pg_walinspect; @@ -134,6 +162,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes @@ -143,6 +173,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; @@ -152,6 +184,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up diff --git a/contrib/pg_walinspect/t/001_timeline.pl b/contrib/pg_walinspect/t/001_timeline.pl new file mode 100644 index 00000000000..1abf7131bb7 --- /dev/null +++ b/contrib/pg_walinspect/t/001_timeline.pl @@ -0,0 +1,62 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test that pg_get_wal_files() follows the current timeline's history across +# a promotion. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +sub test_timeline_history +{ + my ($suffix, $archive_mode) = @_; + my $primary = PostgreSQL::Test::Cluster->new("primary_$suffix"); + my $standby = PostgreSQL::Test::Cluster->new("standby_$suffix"); + + $primary->init(allows_streaming => 1); + $primary->append_conf('postgresql.conf', "autovacuum = off\n"); + if ($archive_mode) + { + $primary->append_conf( + 'postgresql.conf', + "archive_mode = on\narchive_command = 'true'\n"); + } + $primary->start; + $primary->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); + + my $backup_name = "backup_$suffix"; + $primary->backup($backup_name); + $standby->init_from_backup($primary, $backup_name, has_streaming => 1); + $standby->start; + + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (1)'); + my $start_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (2)'); + $primary->wait_for_catchup($standby); + + $standby->promote; + $standby->safe_psql('postgres', 'INSERT INTO test_table VALUES (3)'); + + my $result = $standby->safe_psql( + 'postgres', qq{ +SELECT count(DISTINCT left(wal_file, 8)) = 2, + min(left(wal_file, 8)) = '00000001', + max(left(wal_file, 8)) = '00000002' +FROM pg_get_wal_files('$start_lsn', pg_current_wal_lsn())}); + is($result, 't|t|t', + "WAL files span the timeline switch with archive_mode=$archive_mode"); + + $standby->stop; + $primary->stop; + return; +} + +test_timeline_history('archive_off', 0); +test_timeline_history('archive_on', 1); + +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 79c3ead40bc..b4ecae8cfb1 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -34,9 +34,8 @@ </note> <tip> <para> - All of the <filename>pg_walinspect</filename> functions that show - information about records that fall within a certain LSN range are - permissive about accepting <replaceable>end_lsn</replaceable> + All of the <filename>pg_walinspect</filename> functions that accept an LSN + range are permissive about accepting <replaceable>end_lsn</replaceable> arguments that are after the server's current LSN. Using an <replaceable>end_lsn</replaceable> <quote>from the future</quote> will not raise an error. @@ -60,6 +59,35 @@ <title>General Functions</title> <variablelist> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-files"> + <term> + <function>pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record</function> + </term> + + <listitem> + <para> + If <replaceable>end_lsn</replaceable> is specified and differs from + <replaceable>start_lsn</replaceable>, returns each retained WAL segment + file that intersects the half-open WAL range + [<replaceable>start_lsn</replaceable>, + <replaceable>end_lsn</replaceable>). If + <replaceable>end_lsn</replaceable> is omitted, is null, or equals + <replaceable>start_lsn</replaceable>, returns the segment containing + <replaceable>start_lsn</replaceable>. The + <structfield>wal_file</structfield> column contains the file name, while + <structfield>segment_start_lsn</structfield> and + <structfield>segment_end_lsn</structfield> contain the complete segment + boundaries rather than boundaries clipped to the requested range. + </para> + <para> + File names use the timeline valid for each segment in the current + timeline's history. This allows a returned range to cross a timeline + switch. The function reports an error if any required segment file is + not currently retained in <filename>pg_wal</filename>. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 656f1f60862..1f1bcf8d527 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3434,6 +3434,7 @@ WalSummarizerData WalSummaryFile WalSummaryIO WalTimeSample +WalTimeSegment WalUsage WalWriteMethod WalWriteMethodOps -- 2.50.1 (Apple Git-155) =
v2-0002-pg_walinspect-add-function-to-locate-WAL-by-time.patch (application/octet-stream, 54.5K) download From d939eef1f5ddfa2a122024b6d7371b4a67a6564f Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v2 2/2] pg_walinspect: add function to locate WAL by time Add pg_get_wal_location_at_time(), which uses timestamped WAL records to locate a useful WAL range around a requested wall-clock time. Use an approximate binary search over retained segments and allow callers to widen the requested interval when timestamp reordering makes the initial result insufficient. Move WAL record timestamp extraction from recovery into xlogreader so pg_walinspect can recognize the same timestamp-bearing record types without duplicating that logic. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- .../pg_walinspect/expected/oldextversions.out | 9 +- .../pg_walinspect/expected/pg_walinspect.out | 161 +++++++ contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 21 + contrib/pg_walinspect/pg_walinspect.c | 422 ++++++++++++++++++ contrib/pg_walinspect/sql/pg_walinspect.sql | 118 +++++ contrib/pg_walinspect/t/002_time_search.pl | 51 +++ doc/src/sgml/pgwalinspect.sgml | 107 +++++ src/backend/access/transam/xlogreader.c | 36 ++ src/backend/access/transam/xlogrecovery.c | 48 +- src/include/access/xlogreader.h | 3 + src/tools/pgindent/typedefs.list | 1 + 12 files changed, 932 insertions(+), 46 deletions(-) create mode 100644 contrib/pg_walinspect/t/002_time_search.pl diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 311017d66ca..6228ad9ef39 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -51,15 +51,16 @@ ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; -- List what the current version contains. \dx+ pg_walinspect - Objects in extension "pg_walinspect" - Object description -------------------------------------------------------- + Objects in extension "pg_walinspect" + Object description +---------------------------------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) function pg_get_wal_files(pg_lsn,pg_lsn) + function pg_get_wal_location_at_time(timestamp with time zone,interval,interval) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(5 rows) +(6 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index 34864c65ec7..5933c80071d 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -139,6 +139,142 @@ FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +SELECT start_timestamp >= :'wal_time_lower'::timestamptz + AS start_ok, + end_timestamp > :'wal_time_lower'::timestamptz AND + end_timestamp <= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- A single matching timestamped record is returned for both boundaries. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +SELECT location.start_timestamp = location.end_timestamp AS timestamp_ok, + location.start_lsn = location.end_lsn AS lsn_ok, + info.record_type = 'COMMIT' AS record_ok +FROM pg_get_wal_location_at_time( + :'single_time_lower'::timestamptz + interval '1 microsecond', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz - interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS info; + timestamp_ok | lsn_ok | record_ok +--------------+--------+----------- + t | t | t +(1 row) + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + timestamped_ok +---------------- + t +(1 row) + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | end_ok +----------+-------- + t | t +(1 row) + +-- A future upper bound uses a timestamped WAL record at or before now. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp >= :'restore_time_upper'::timestamptz AND + location.start_timestamp <= :'current_time_target'::timestamptz + AS start_ok, + location.end_timestamp >= :'current_time_target'::timestamptz AND + location.end_timestamp <= clock_timestamp() AS capped_end_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + end_info.record_type = 'COMMIT' AS real_record_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | capped_end_ok | lsn_ok | real_record_ok +----------+---------------+--------+---------------- + t | t | t | t +(1 row) + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; + defaults_ok +------------- + t +(1 row) + +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +ERROR: before interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +ERROR: after interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +ERROR: could not locate a WAL time boundary +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); +ERROR: requested time follows the available WAL range +-- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset @@ -231,6 +367,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no has_function_privilege @@ -268,6 +411,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -285,6 +435,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -315,6 +467,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -330,6 +489,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 6f8b4a6359a..79e94e33e4d 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -41,6 +41,7 @@ tests += { 'tap': { 'tests': [ 't/001_timeline.pl', + 't/002_time_search.pl', ], }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql index 9cc9523b4dc..f72d99a3ceb 100644 --- a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -3,6 +3,27 @@ -- complain if script is sourced in psql, rather than via ALTER EXTENSION \echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit +-- +-- pg_get_wal_location_at_time() +-- +CREATE FUNCTION pg_get_wal_location_at_time( + IN target_time timestamptz, + IN before interval DEFAULT '1 minute', + IN after interval DEFAULT '1 minute', + OUT start_timestamp timestamptz, + OUT start_lsn pg_lsn, + OUT end_timestamp timestamptz, + OUT end_lsn pg_lsn +) +AS 'MODULE_PATHNAME', 'pg_get_wal_location_at_time' +LANGUAGE C STRICT PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) + TO pg_read_server_files; + -- -- pg_get_wal_files() -- diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index 42f366bdb4e..0511bb591e7 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -12,6 +12,8 @@ */ #include "postgres.h" +#include <sys/stat.h> + #include "access/htup_details.h" #include "access/timeline.h" #include "access/xlog.h" @@ -26,7 +28,9 @@ #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgrprotos.h" #include "utils/pg_lsn.h" +#include "utils/timestamp.h" #include "utils/tuplestore.h" /* @@ -41,6 +45,7 @@ PG_MODULE_MAGIC_EXT( PG_FUNCTION_INFO_V1(pg_get_wal_block_info); PG_FUNCTION_INFO_V1(pg_get_wal_files); +PG_FUNCTION_INFO_V1(pg_get_wal_location_at_time); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -71,15 +76,51 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Selected WAL position and timestamp for one time boundary. */ +typedef struct WalTimeBoundary +{ + bool found; /* matching record was found */ + TimestampTz time; /* record's local event time */ + XLogRecPtr lsn; /* record's start LSN */ + int segment_index; /* index of the segment containing the record */ +} WalTimeBoundary; + /* Metadata for one retained segment in the current timeline's history. */ typedef struct WalTimeSegment { XLogSegNo segno; /* segment number, used for WAL ordering */ TimeLineID tli; /* timeline containing this segment */ + bool scanned; /* segment has already been decoded */ + bool has_timestamp; /* segment contains timestamped records */ + TimestampTz min_time; /* minimum record timestamp in segment */ + TimestampTz max_time; /* maximum record timestamp in segment */ + WalTimeBoundary lower; /* best lower anchor in segment */ + WalTimeBoundary upper; /* best upper anchor in segment */ } WalTimeSegment; static int wal_time_segment_cmp(const void *a, const void *b); static WalTimeSegment *GetWalTimeSegments(int *nsegments); +static int FindWalTimeCandidate(WalTimeSegment *segments, + int *search_segments, + int *nsearch_segments, + TimestampTz target_time, + bool search_left, + TimestampTz wanted_start, + TimestampTz wanted_end, + int *candidate_position, + int *nscanned); +static void ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, + TimestampTz wanted_end); +static void MergeWalTimeSegment(WalTimeSegment *segment, + WalTimeBoundary *lower, + WalTimeBoundary *upper); +static void ProbeLowerBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); +static void ProbeUpperBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); /* * Return the LSN up to which the server has WAL, and optionally its timeline. @@ -223,6 +264,44 @@ wal_time_segment_cmp(const void *a, const void *b) return 0; } +/* + * Probe a timestamped record at or after a requested time boundary. Among + * records examined so far, prefer the earliest time, breaking ties in favor + * of the earlier WAL record. + */ +static void +ProbeLowerBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time < boundary->time || + (time == boundary->time && lsn < boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + +/* + * Probe a timestamped record at or before a requested time boundary. Among + * records examined so far, prefer the latest time, breaking ties in favor of + * the later WAL record. + */ +static void +ProbeUpperBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time > boundary->time || + (time == boundary->time && lsn > boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + /* * Get the WAL segments that form the history of the server's current * timeline. Other timelines may have files with the same segment number in @@ -297,6 +376,140 @@ GetWalTimeSegments(int *nsegments) return segments; } +/* + * Approximately locate the segment containing target_time by binary search. + * Record timestamps are expected to generally increase with WAL position. + * Remove segments without timestamped records from the search candidates, + * since they provide no value with which to choose either half. + */ +static int +FindWalTimeCandidate(WalTimeSegment *segments, int *search_segments, + int *nsearch_segments, + TimestampTz target_time, bool search_left, + TimestampTz wanted_start, TimestampTz wanted_end, + int *candidate_position, int *nscanned) +{ + int left = 0; + int right = *nsearch_segments - 1; + int result; + + while (left <= right) + { + int middle = left + (right - left) / 2; + int segment_index = search_segments[middle]; + WalTimeSegment *segment = &segments[segment_index]; + + if (!segment->scanned) + { + ScanWalTimeSegment(segment, segment_index, wanted_start, wanted_end); + (*nscanned)++; + } + + if (!segment->has_timestamp) + { + memmove(&search_segments[middle], &search_segments[middle + 1], + (*nsearch_segments - middle - 1) * sizeof(int)); + (*nsearch_segments)--; + right--; + continue; + } + else if (target_time < segment->min_time) + right = middle - 1; + else if (target_time > segment->max_time) + left = middle + 1; + else + { + if (candidate_position != NULL) + *candidate_position = middle; + return search_segments[middle]; + } + } + + if (*nsearch_segments == 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("No retained WAL segment contains a timestamped record.")); + + if (search_left) + result = Max(right, 0); + else + result = Min(left, *nsearch_segments - 1); + + if (candidate_position != NULL) + *candidate_position = result; + return search_segments[result]; +} + +/* + * Inspect timestamped records beginning in one segment and save its time + * range and best boundary candidates. XLogReadRecord() may read the following + * segment to assemble a record crossing the boundary, but that record is + * attributed to the segment containing its starting LSN. + */ +static void +ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, TimestampTz wanted_end) +{ + XLogRecPtr seg_start = segment->segno * wal_segment_size; + XLogRecPtr seg_end = seg_start + wal_segment_size; + XLogReaderState *xlogreader; + + Assert(!segment->scanned); + segment->scanned = true; + xlogreader = InitXLogReaderState(seg_start); + + while (ReadNextXLogRecord(xlogreader)) + { + TimestampTz record_time; + + if (xlogreader->ReadRecPtr >= seg_end) + break; + + if (!GetXLogRecordTimestamp(xlogreader, &record_time)) + continue; + + if (!segment->has_timestamp) + { + segment->has_timestamp = true; + segment->min_time = segment->max_time = record_time; + } + else + { + segment->min_time = Min(segment->min_time, record_time); + segment->max_time = Max(segment->max_time, record_time); + } + + if (record_time >= wanted_start) + ProbeLowerBoundary(&segment->lower, record_time, + xlogreader->ReadRecPtr, segment_index); + + if (record_time <= wanted_end) + ProbeUpperBoundary(&segment->upper, record_time, + xlogreader->ReadRecPtr, segment_index); + + CHECK_FOR_INTERRUPTS(); + } + + pfree(xlogreader->private_data); + XLogReaderFree(xlogreader); +} + +/* Merge one segment's boundary candidates into the result. */ +static void +MergeWalTimeSegment(WalTimeSegment *segment, WalTimeBoundary *lower, + WalTimeBoundary *upper) +{ + Assert(segment->scanned); + + if (lower != NULL && segment->lower.found) + ProbeLowerBoundary(lower, segment->lower.time, segment->lower.lsn, + segment->lower.segment_index); + if (upper != NULL && segment->upper.found) + ProbeUpperBoundary(upper, segment->upper.time, segment->upper.lsn, + segment->upper.segment_index); +} + /* * Output values that make up a row describing caller's WAL record. * @@ -623,6 +836,215 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * Locate a WAL region around a wall-clock time range using timestamped WAL + * records. + * + * Locate each boundary by an approximate binary search over retained WAL + * segments, assuming that record timestamps generally increase with WAL + * position. Also examine one preceding segment for the lower boundary and + * one following segment for the upper boundary to allow for limited timestamp + * reordering. The lower boundary uses the earliest timestamped record at or + * after the requested start time, while the upper boundary uses the latest + * one at or before the requested end time. A future upper bound is first + * capped at the current time. On success, the boundaries form a forward WAL + * range, but are not necessarily the globally closest timestamped records to + * the requested bounds. + */ +Datum +pg_get_wal_location_at_time(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_LOCATION_AT_TIME_COLS 4 + TimestampTz target_time = PG_GETARG_TIMESTAMPTZ(0); + Interval *before = PG_GETARG_INTERVAL_P(1); + Interval *after = PG_GETARG_INTERVAL_P(2); + Interval zero = {0}; + Interval one_day = {.day = 1}; + TimestampTz wanted_start; + TimestampTz wanted_end; + TimestampTz current_time; + WalTimeSegment *segments; + WalTimeBoundary lower = {0}; + WalTimeBoundary upper = {0}; + Datum values[PG_GET_WAL_LOCATION_AT_TIME_COLS]; + bool nulls[PG_GET_WAL_LOCATION_AT_TIME_COLS] = {0}; + TupleDesc tupdesc; + HeapTuple tuple; + int nsegments; + int *search_segments; + int nsearch_segments; + int lower_candidate; + int upper_candidate; + int lower_candidate_position; + int nupper_search_segments; + int lower_start; + int upper_end; + int nscanned = 0; + char lower_fname[MAXFNAMELEN]; + char upper_fname[MAXFNAMELEN]; + + if (TIMESTAMP_NOT_FINITE(target_time)) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("target time must be finite")); + if (INTERVAL_NOT_FINITE(before) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must not exceed one day")); + if (INTERVAL_NOT_FINITE(after) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must not exceed one day")); + + /* The timestamp functions provide the normal overflow checks. */ + wanted_start = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_mi_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(before))); + wanted_end = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_pl_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(after))); + current_time = GetCurrentTimestamp(); + + /* Cap a future upper bound at the time currently available. */ + if (wanted_end > current_time) + { + if (wanted_start >= current_time) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("The requested lower bound is not earlier than the current time.")); + + wanted_end = current_time; + } + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + segments = GetWalTimeSegments(&nsegments); + search_segments = palloc_array(int, nsegments); + for (int i = 0; i < nsegments; i++) + search_segments[i] = i; + nsearch_segments = nsegments; + lower_candidate = FindWalTimeCandidate(segments, search_segments, + &nsearch_segments, wanted_start, + false, wanted_start, wanted_end, + &lower_candidate_position, + &nscanned); + + /* Search for the later upper boundary from the lower candidate onward. */ + nupper_search_segments = nsearch_segments - lower_candidate_position; + upper_candidate = FindWalTimeCandidate(segments, + search_segments + lower_candidate_position, + &nupper_search_segments, wanted_end, + true, wanted_start, wanted_end, + NULL, + &nscanned); + XLogFileName(lower_fname, segments[lower_candidate].tli, + segments[lower_candidate].segno, wal_segment_size); + XLogFileName(upper_fname, segments[upper_candidate].tli, + segments[upper_candidate].segno, wal_segment_size); + ereport(DEBUG1, + errmsg_internal("WAL time search selected lower segment %s and upper segment %s from %d retained segments", + lower_fname, upper_fname, + nsegments)); + + /* + * Allow for limited timestamp reordering by examining one preceding + * segment for the lower boundary and one following segment for the upper + * boundary. Do not extend across a missing segment. + */ + lower_start = lower_candidate; + if (lower_start > 0 && + segments[lower_start - 1].segno + 1 == segments[lower_start].segno) + lower_start--; + upper_end = upper_candidate; + if (upper_end + 1 < nsegments && + segments[upper_end].segno + 1 == segments[upper_end + 1].segno) + upper_end++; + + for (int i = lower_start; i <= lower_candidate; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], &lower, NULL); + } + for (int i = upper_candidate; i <= upper_end; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], NULL, &upper); + } + + ereport(DEBUG1, + errmsg_internal("WAL time search decoded %d of %d retained segments", + nscanned, nsegments)); + + if (!lower.found) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("The WAL search did not find a timestamped record at or after the requested lower bound.")); + if (!upper.found) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("The WAL search did not find a timestamped record at or before the requested upper bound.")); + + if (lower.lsn > upper.lsn) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("could not find a valid WAL range for the requested time window"), + errdetail("The matching lower boundary follows the upper boundary in WAL.")); + + Assert(lower.segment_index >= 0 && lower.segment_index < nsegments); + Assert(upper.segment_index >= 0 && upper.segment_index < nsegments); + for (int i = lower.segment_index + 1; i <= upper.segment_index; i++) + { + if (segments[i - 1].segno + 1 != segments[i].segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) (segments[i - 1].segno + 1))); + } + + values[0] = TimestampTzGetDatum(lower.time); + values[1] = LSNGetDatum(lower.lsn); + values[2] = TimestampTzGetDatum(upper.time); + values[3] = LSNGetDatum(upper.lsn); + + pfree(segments); + tuple = heap_form_tuple(tupdesc, values, nulls); + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +#undef PG_GET_WAL_LOCATION_AT_TIME_COLS +} + /* * List the retained WAL segment files intersecting [start_lsn, end_lsn), or * the segment containing start_lsn when end_lsn is omitted or equal to it. diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 3176a17d855..dbc3240ae30 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -75,6 +75,114 @@ SELECT count(*) = 1 AS future_end_ok FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== + +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset + +SELECT start_timestamp >= :'wal_time_lower'::timestamptz + AS start_ok, + end_timestamp > :'wal_time_lower'::timestamptz AND + end_timestamp <= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + +-- A single matching timestamped record is returned for both boundaries. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +SELECT location.start_timestamp = location.end_timestamp AS timestamp_ok, + location.start_lsn = location.end_lsn AS lsn_ok, + info.record_type = 'COMMIT' AS record_ok +FROM pg_get_wal_location_at_time( + :'single_time_lower'::timestamptz + interval '1 microsecond', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz - interval '1 microsecond') AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS info; + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset + +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); + +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +-- A future upper bound uses a timestamped WAL record at or before now. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp >= :'restore_time_upper'::timestamptz AND + location.start_timestamp <= :'current_time_target'::timestamptz + AS start_ok, + location.end_timestamp >= :'current_time_target'::timestamptz AND + location.end_timestamp <= clock_timestamp() AS capped_end_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + end_info.record_type = 'COMMIT' AS real_record_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -134,6 +242,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no @@ -148,6 +258,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -162,6 +274,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; @@ -173,6 +287,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -184,6 +300,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; diff --git a/contrib/pg_walinspect/t/002_time_search.pl b/contrib/pg_walinspect/t/002_time_search.pl new file mode 100644 index 00000000000..85ed7fb7bf4 --- /dev/null +++ b/contrib/pg_walinspect/t/002_time_search.pl @@ -0,0 +1,51 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test the bounded binary search for WAL time boundaries across segments. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $node = PostgreSQL::Test::Cluster->new('time_search'); +$node->init(extra => ['--wal-segsize=1']); +$node->append_conf('postgresql.conf', "autovacuum = off\n"); +$node->start; +$node->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); +$node->safe_psql('postgres', 'SELECT pg_switch_wal()'); + +my ($lower_time, $upper_time); +for my $i (0 .. 8) +{ + $node->safe_psql('postgres', "INSERT INTO test_table VALUES ($i)"); + $lower_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 3; + $upper_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 5; + $node->safe_psql('postgres', 'SELECT pg_switch_wal()') if $i < 8; +} + +my $log_offset = -s $node->logfile; +my $result = $node->safe_psql( + 'postgres', qq{ +SET log_min_messages = debug1; +SELECT start_timestamp >= '$lower_time'::timestamptz - interval '1 microsecond', + end_timestamp > '$lower_time'::timestamptz, + end_timestamp <= '$upper_time'::timestamptz, + start_lsn < end_lsn, + pg_walfile_name(start_lsn) <> pg_walfile_name(end_lsn) +FROM pg_get_wal_location_at_time('$lower_time', + interval '1 microsecond', + '$upper_time'::timestamptz - + '$lower_time'::timestamptz)}); +is($result, 't|t|t|t|t', 'binary search finds boundaries across WAL segments'); + +my $log = slurp_file($node->logfile, $log_offset); +$log =~ /WAL time search decoded (\d+) of (\d+) retained segments/ + or die "could not find WAL time search statistics in server log"; +cmp_ok($1, '<', $2, 'binary search does not decode all retained WAL'); + +$node->stop; +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index b4ecae8cfb1..e00beb7d657 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -88,6 +88,113 @@ </listitem> </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-location-at-time"> + <term> + <function>pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute') returns record</function> + </term> + + <listitem> + <para> + Locates a nearby WAL region using records containing local event + timestamps as historical time anchors. The requested time range extends + from <replaceable>target_time</replaceable> minus + <replaceable>before</replaceable> through + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable>. It returns + <structfield>start_timestamp</structfield>, + <structfield>start_lsn</structfield>, + <structfield>end_timestamp</structfield>, and + <structfield>end_lsn</structfield>. On successful return, the following + relationships hold: +<programlisting> +start_timestamp &gt;= target_time - before +end_timestamp &lt;= target_time + after +start_lsn &lt;= end_lsn +</programlisting> + The start boundary is the earliest timestamped record found at or after + the requested start time, and the end boundary is the latest timestamped + record found at or before the requested end time. If + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> is later than the current time, the + requested upper bound is first capped at the current time. The function + does not construct an end boundary from the current WAL position. Thus, + both returned boundaries come from actual timestamped WAL records. If + only one timestamped record matches the requested time range, both + boundaries identify that record and the returned timestamps and LSNs + are equal. Both interval arguments must be greater than zero and not + greater than one day. + </para> + <para> + Most WAL records do not contain a wall-clock timestamp. Historical + anchors can be transaction <literal>COMMIT</literal>, + <literal>COMMIT PREPARED</literal>, <literal>ABORT</literal>, + or <literal>ABORT PREPARED</literal> records, as well as restore point + records. These are the WAL record types from which PostgreSQL recovery + can extract timestamps. Their timestamps describe the corresponding + event; they do not assign timestamps to heap, index, or other records. + </para> + <para> + In particular, a commit timestamp describes when a transaction + committed, not when its earlier changes generated WAL. For example, a + transaction might update a row at 14:00 and commit at 15:00; the heap + WAL was generated around 14:00, while the commit record is timestamped + 15:00. The returned LSNs therefore do not identify the exact generation + time of heap, index, or other WAL records, nor do they guarantee that + every record generated during the requested wall-clock range lies + between the returned LSNs. On a cluster with little timestamped + activity, the returned timestamps can be far from the requested bounds. + </para> + <para> + The function assumes that WAL record timestamps generally increase with + WAL position and uses an approximate binary search to locate each time + boundary. WAL record timestamps are not guaranteed to follow WAL order, + for example because of concurrent activity, clock adjustments, or + delayed WAL insertion. Such timestamp reordering can cause relevant + records to fall outside the returned range. Therefore, the returned + records are not guaranteed to be the globally closest timestamped + records to either requested time bound. If the returned range does not + contain the records of interest, the user can widen the search range and + repeat the search. If the matching lower boundary follows the upper + boundary in WAL, the function reports an error rather than returning a + range with the LSNs reversed. + </para> + <para> + Only WAL that is currently retained in <filename>pg_wal</filename> can + be searched. Unlike the other <application>pg_walinspect</application> + functions, this function may inspect retained WAL from the current + timeline's history as well as the current timeline. The function + reports an error when either boundary cannot be found by the bounded + search or when the resulting range would cross a missing WAL segment. + </para> + <para> + For example, a DBA can locate WAL around activity near a given time and + then inspect the returned range: +<programlisting> +SELECT * +FROM pg_get_wal_location_at_time( + '2026-08-01 14:35:00+08', + before =&gt; '10 minutes', + after =&gt; '10 minutes'); + +SELECT * +FROM pg_get_wal_files('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); + +SELECT * +FROM pg_get_wal_records_info('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); +</programlisting> + The corresponding retained WAL files can be obtained with + <function>pg_get_wal_files</function>. To widen the search range, + increase <replaceable>before</replaceable> if the relevant activity + might have occurred earlier, or increase + <replaceable>after</replaceable> if it might have occurred later, and + repeat the search. Failure to find a record in the first returned range + does not necessarily mean that the record is absent from retained WAL. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7db7c273b0c..02f49f64b3b 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -26,6 +26,7 @@ #endif #include "access/transam.h" +#include "access/xact.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" #include "access/xlogrecord.h" @@ -65,6 +66,41 @@ static void WALOpenSegmentInit(WALOpenSegment *seg, WALSegmentContext *segcxt, */ #define DEFAULT_DECODE_BUFFER_SIZE (64 * 1024) +/* + * Extract timestamp from WAL record. + * + * If the record contains a timestamp, returns true, and saves the timestamp + * in *recordXtime. If the record type has no timestamp, returns false. + * Currently, only transaction commit/abort records and restore points contain + * timestamps. + */ +bool +GetXLogRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) +{ + uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; + uint8 xact_info = info & XLOG_XACT_OPMASK; + uint8 rmid = XLogRecGetRmid(record); + + if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) + { + *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || + xact_info == XLOG_XACT_COMMIT_PREPARED)) + { + *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || + xact_info == XLOG_XACT_ABORT_PREPARED)) + { + *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; + return true; + } + return false; +} + /* * Construct a string in state->errormsg_buf explaining what's wrong with * the current record being read. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 54aaec9529f..06a5b10d092 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -356,7 +356,6 @@ static void xlog_outrec(StringInfo buf, XLogReaderState *record); static void xlog_block_info(StringInfo buf, XLogReaderState *record); static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, TimeLineID replayTLI); -static bool getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime); static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); @@ -2405,41 +2404,6 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, } -/* - * Extract timestamp from WAL record. - * - * If the record contains a timestamp, returns true, and saves the timestamp - * in *recordXtime. If the record type has no timestamp, returns false. - * Currently, only transaction commit/abort records and restore points contain - * timestamps. - */ -static bool -getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) -{ - uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; - uint8 xact_info = info & XLOG_XACT_OPMASK; - uint8 rmid = XLogRecGetRmid(record); - - if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) - { - *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || - xact_info == XLOG_XACT_COMMIT_PREPARED)) - { - *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || - xact_info == XLOG_XACT_ABORT_PREPARED)) - { - *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; - return true; - } - return false; -} - /* * Checks whether the current buffer page and backup page stored in the * WAL record are consistent or not. Before comparing the two pages, a @@ -2663,10 +2627,10 @@ recoveryStopsBefore(XLogReaderState *record) /* * Note: we must fetch recordXtime regardless of recoveryTarget setting. - * We don't expect getRecordTimestamp ever to fail, since we already know - * this is a commit or abort record; but test its result anyway. + * We don't expect GetXLogRecordTimestamp ever to fail, since we already + * know this is a commit or abort record; but test its result anyway. */ - if (getRecordTimestamp(record, &recordXtime) && + if (GetXLogRecordTimestamp(record, &recordXtime) && recoveryTarget == RECOVERY_TARGET_TIME) { /* @@ -2747,7 +2711,7 @@ recoveryStopsAfter(XLogReaderState *record) recoveryStopAfter = true; recoveryStopXid = InvalidTransactionId; recoveryStopLSN = InvalidXLogRecPtr; - (void) getRecordTimestamp(record, &recoveryStopTime); + (void) GetXLogRecordTimestamp(record, &recoveryStopTime); strlcpy(recoveryStopName, recordRestorePointData->rp_name, MAXFNAMELEN); ereport(LOG, @@ -2787,7 +2751,7 @@ recoveryStopsAfter(XLogReaderState *record) TransactionId recordXid; /* Update the last applied transaction timestamp */ - if (getRecordTimestamp(record, &recordXtime)) + if (GetXLogRecordTimestamp(record, &recordXtime)) SetLatestXTime(recordXtime); /* Extract the XID of the committed/aborted transaction */ @@ -3006,7 +2970,7 @@ recoveryApplyDelay(XLogReaderState *record) xact_info != XLOG_XACT_COMMIT_PREPARED) return false; - if (!getRecordTimestamp(record, &xtime)) + if (!GetXLogRecordTimestamp(record, &xtime)) return false; delayUntil = TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 4a9a687e879..3016ae4d5da 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -39,6 +39,7 @@ #endif #include "access/xlogrecord.h" +#include "datatype/timestamp.h" #include "storage/buf.h" /* WALOpenSegment represents a WAL segment being read. */ @@ -440,5 +441,7 @@ extern bool XLogRecGetBlockTagExtended(XLogReaderState *record, uint8 block_id, RelFileLocator *rlocator, ForkNumber *forknum, BlockNumber *blknum, Buffer *prefetch_buffer); +extern bool GetXLogRecordTimestamp(XLogReaderState *record, + TimestampTz *recordXtime); #endif /* XLOGREADER_H */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 1f1bcf8d527..c85279cf423 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3433,6 +3433,7 @@ WalSndState WalSummarizerData WalSummaryFile WalSummaryIO +WalTimeBoundary WalTimeSample WalTimeSegment WalUsage -- 2.50.1 (Apple Git-155) =
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-06 23:11  surya poondla <suryapoondla4@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: surya poondla @ 2026-10-06 23:11 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi Chao,

Thanks for the quick patch on v2, and for splitting the patch. I checked
the changes against my earlier comments, point 1 is fixed (end_lsn is now
capped before point_lookup is set), also point 2 (the invalid startLSN
message), the timeline TAP test (with archive_mode both off and on), the
test robustness changes, the move of GetXLogRecordTimestamp() to
xlogreader.c, and all of the minor items.

v2 applies cleanly to master (10b6e2a3d66) and builds without warnings, and
the regression and TAP tests pass.

0001 (pg_get_wal_files) is close. One edge case from reading the code,
GetWalTimeSegments() skips segments that start at or after the current LSN.
So if the flush LSN sits exactly on a segment boundary, as on an idle
server right after pg_switch_wal(), pg_get_wal_files(pg_current_wal_lsn())
becomes
a point lookup on a filtered-out segment and reports "WAL segment needed
for the requested range is missing".

For 0002, I think the change of boundary semantics breaks the main use
case. Since v2, start_lsn is the earliest timestamped record found at or
after the
lower bound, and end_lsn is the latest found at or before the upper bound.
But a transaction's data records come before its commit record in WAL.
So when the commit of the transaction of interest is the first timestamped
record in the window, start_lsn is that commit, and the change itself falls
outside the returned range.

I reproduced this with a scratch TAP test: autovacuum off, and no other
commits within two seconds of the window.

  SELECT clock_timestamp() AS window_start \gset
  BEGIN;
  SELECT pg_current_wal_insert_lsn() AS delete_lsn \gset
  DELETE FROM important WHERE id = 5;
  SELECT pg_sleep(0.2);
  COMMIT;
  SELECT clock_timestamp() AS window_end \gset

  SELECT l.start_lsn, l.end_lsn, :'delete_lsn'::pg_lsn AS delete_lsn
  FROM pg_get_wal_location_at_time(:'window_start',
                                   interval '500 milliseconds',
                                   :'window_end'::timestamptz -
                                     :'window_start'::timestamptz) AS l;

   start_lsn  |  end_lsn   | delete_lsn
  ------------+------------+------------
   0/017EEF78 | 0/017EEF78 | 0/017EEF40

  The DELETE ran inside the requested window, and its WAL begins at
  0/017EEF40, the insert position taken just before it. Its commit, 56 bytes
  later, is the only timestamped record in the window, so it's returned as
  both boundaries. Following the documented workflow,
  pg_get_wal_records_info(start_lsn, end_lsn) then returns no rows at all,
  because that function only emits records that end at or before end_lsn. So
  the user sees neither the DELETE nor its commit. On a quiet system, which
is
  when someone is most likely to be investigating a single mistaken
statement,
  this is the usual outcome, not a corner case.

Outer anchors (the latest timestamped record at or before the lower bound,
and the earliest at or after the upper bound) avoid this, but only with a
fallback for the end. In this reproducer nothing commits after the window,
so by my reading v1 would have failed with "requested time follows the
available WAL range" instead. pg_walinspect already treats the current WAL
position as a valid end (ValidateInputLSNs() caps end_lsn to it, and the
docs suggest FFFFFFFF/FFFFFFFF for exactly that), so falling back to the
current flush/replay LSN, with end_timestamp NULL, seems consistent with the
rest of the module. It would also cover the quiet-system case from my first
review.

I don't think the binary search requires inner anchors. It can locate the
segment for each bound and then take the outer record from that segment, or
from the nearest earlier (or later) segment that has a timestamped record.
Outer anchors still won't cover a transaction that began before the lower
anchor, but the docs already say as much, and it is a much narrower case.

Separately, reading the code, a window that contains no timestamped record,
but has some on both sides, currently ends in "could not find a valid WAL
range for the requested time window" with the detail "The matching lower
boundary follows the upper boundary in WAL". That reads like a WAL ordering
problem, when the real cause is that nothing in the window is timestamped.
With outer anchors this case would just return a range.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-08 06:13  Chao Li <li.evan.chao@gmail.com>
  parent: surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Chao Li @ 2026-10-08 06:13 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>



> On Oct 7, 2026, at 07:11, surya poondla <suryapoondla4@gmail.com> wrote:
> 
> Hi Chao,
> 
> Thanks for the quick patch on v2, and for splitting the patch. I checked the changes against my earlier comments, point 1 is fixed (end_lsn is now capped before point_lookup is set), also point 2 (the invalid startLSN message), the timeline TAP test (with archive_mode both off and on), the test robustness changes, the move of GetXLogRecordTimestamp() to
> xlogreader.c, and all of the minor items. 
> 
> v2 applies cleanly to master (10b6e2a3d66) and builds without warnings, and the regression and TAP tests pass.

Thanks again for the follow up review and tests.

> 
> 0001 (pg_get_wal_files) is close. One edge case from reading the code, GetWalTimeSegments() skips segments that start at or after the current LSN.
> So if the flush LSN sits exactly on a segment boundary, as on an idle server right after pg_switch_wal(), pg_get_wal_files(pg_current_wal_lsn()) becomes
> a point lookup on a filtered-out segment and reports "WAL segment needed for the requested range is missing".

Good catch. Fixed in v3.

> 
> For 0002, I think the change of boundary semantics breaks the main use case. Since v2, start_lsn is the earliest timestamped record found at or after the
> lower bound, and end_lsn is the latest found at or before the upper bound. But a transaction's data records come before its commit record in WAL. 
> So when the commit of the transaction of interest is the first timestamped record in the window, start_lsn is that commit, and the change itself falls
> outside the returned range.
> 
> I reproduced this with a scratch TAP test: autovacuum off, and no other commits within two seconds of the window.
> 
>   SELECT clock_timestamp() AS window_start \gset
>   BEGIN;
>   SELECT pg_current_wal_insert_lsn() AS delete_lsn \gset
>   DELETE FROM important WHERE id = 5;
>   SELECT pg_sleep(0.2);
>   COMMIT;
>   SELECT clock_timestamp() AS window_end \gset
> 
>   SELECT l.start_lsn, l.end_lsn, :'delete_lsn'::pg_lsn AS delete_lsn
>   FROM pg_get_wal_location_at_time(:'window_start',
>                                    interval '500 milliseconds',
>                                    :'window_end'::timestamptz -
>                                      :'window_start'::timestamptz) AS l;
> 
>    start_lsn  |  end_lsn   | delete_lsn
>   ------------+------------+------------
>    0/017EEF78 | 0/017EEF78 | 0/017EEF40
> 
>   The DELETE ran inside the requested window, and its WAL begins at
>   0/017EEF40, the insert position taken just before it. Its commit, 56 bytes
>   later, is the only timestamped record in the window, so it's returned as
>   both boundaries. Following the documented workflow,
>   pg_get_wal_records_info(start_lsn, end_lsn) then returns no rows at all,
>   because that function only emits records that end at or before end_lsn. So
>   the user sees neither the DELETE nor its commit. On a quiet system, which is
>   when someone is most likely to be investigating a single mistaken statement,
>   this is the usual outcome, not a corner case.
> 
> Outer anchors (the latest timestamped record at or before the lower bound, and the earliest at or after the upper bound) avoid this, but only with a
> fallback for the end. In this reproducer nothing commits after the window, so by my reading v1 would have failed with "requested time follows the
> available WAL range" instead. pg_walinspect already treats the current WAL position as a valid end (ValidateInputLSNs() caps end_lsn to it, and the
> docs suggest FFFFFFFF/FFFFFFFF for exactly that), so falling back to the current flush/replay LSN, with end_timestamp NULL, seems consistent with the
> rest of the module. It would also cover the quiet-system case from my first review.
> 
> I don't think the binary search requires inner anchors. It can locate the segment for each bound and then take the outer record from that segment, or
> from the nearest earlier (or later) segment that has a timestamped record.
> Outer anchors still won't cover a transaction that began before the lower anchor, but the docs already say as much, and it is a much narrower case.

I was thinking that, since the time range might be rough anyway, users could simply widen the window to search for more records. But your point is also valid. In some cases, such as verifying the function using a small window, using outer anchors is more reasonable.

Yes, outer anchors can still be used with binary search. I have changed v3 to use outer anchors, and your reproducer now gets:
```
evantest=#   SELECT l.start_lsn, l.end_lsn, :'delete_lsn'::pg_lsn AS delete_lsn
evantest-#   FROM pg_get_wal_location_at_time(:'window_start',
evantest(#                                    interval '500 milliseconds',
evantest(#                                    :'window_end'::timestamptz -
evantest(#                                      :'window_start'::timestamptz) AS l;
 start_lsn  |  end_lsn   | delete_lsn
------------+------------+------------
 0/01C50430 | 0/01C504F0 | 0/01C50490
(1 row)
```

> 
> Separately, reading the code, a window that contains no timestamped record,
> but has some on both sides, currently ends in "could not find a valid WAL
> range for the requested time window" with the detail "The matching lower boundary follows the upper boundary in WAL". That reads like a WAL ordering
> problem, when the real cause is that nothing in the window is timestamped.
> With outer anchors this case would just return a range.

Yes. I added a tested for that.

PFA v3 - addressed Surya’s review comments.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments: download all (2 files, .tar.gz)

v3-0001-pg_walinspect-add-function-to-list-WAL-files-by-L.patch (application/octet-stream, 29.0K) download From b4e31927125ec80be770ba4ae77268ef1b89a7b6 Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v3 1/2] pg_walinspect: add function to list WAL files by LSN Add pg_get_wal_files(), which returns the retained WAL segment files intersecting an LSN range, together with each segment's complete LSN boundaries. A point lookup can be requested by omitting end_lsn or making it equal to start_lsn. Choose segment file names from the server's current timeline history so that ranges spanning a timeline switch are handled correctly. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- contrib/pg_walinspect/Makefile | 4 +- .../pg_walinspect/expected/oldextversions.out | 8 +- .../pg_walinspect/expected/pg_walinspect.out | 73 ++++++ contrib/pg_walinspect/meson.build | 6 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 22 ++ contrib/pg_walinspect/pg_walinspect.c | 219 +++++++++++++++++- contrib/pg_walinspect/pg_walinspect.control | 2 +- contrib/pg_walinspect/sql/oldextversions.sql | 5 +- contrib/pg_walinspect/sql/pg_walinspect.sql | 34 +++ contrib/pg_walinspect/t/001_timeline.pl | 73 ++++++ doc/src/sgml/pgwalinspect.sgml | 37 ++- src/tools/pgindent/typedefs.list | 1 + 12 files changed, 465 insertions(+), 19 deletions(-) create mode 100644 contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql create mode 100644 contrib/pg_walinspect/t/001_timeline.pl diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 22090f7716c..b7651231077 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -7,9 +7,11 @@ OBJS = \ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log" EXTENSION = pg_walinspect -DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql +DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql \ + pg_walinspect--1.1--1.2.sql REGRESS = pg_walinspect oldextversions +TAP_TESTS = 1 REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_walinspect/walinspect.conf diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 89953af3d0e..311017d66ca 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -46,18 +46,20 @@ SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; --- List what version 1.1 contains. +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; +-- List what the current version contains. \dx+ pg_walinspect Objects in extension "pg_walinspect" Object description ------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) + function pg_get_wal_files(pg_lsn,pg_lsn) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(4 rows) +(5 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index f955ff5d3c5..34864c65ec7 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -90,6 +90,54 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); t (1 row) +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + file_ok | start_ok | end_ok +---------+----------+-------- + t | t | t +(1 row) + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + one_file +---------- + t +(1 row) + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); + equal_lsn_ok +-------------- + t +(1 row) + +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); + default_end_ok +---------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +ERROR: WAL start LSN must be less than end LSN +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); + future_end_ok +--------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files('0/0', '0/1'); +ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -183,6 +231,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -213,6 +268,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users. GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) @@ -223,6 +285,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -251,6 +315,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) @@ -259,6 +330,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 82ab186c340..6f8b4a6359a 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -20,6 +20,7 @@ install_data( 'pg_walinspect.control', 'pg_walinspect--1.0.sql', 'pg_walinspect--1.0--1.1.sql', + 'pg_walinspect--1.1--1.2.sql', kwargs: contrib_data_args, ) @@ -37,4 +38,9 @@ tests += { 'regress_args': ['--temp-config', files('walinspect.conf')], 'runningcheck': false, }, + 'tap': { + 'tests': [ + 't/001_timeline.pl', + ], + }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql new file mode 100644 index 00000000000..9cc9523b4dc --- /dev/null +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -0,0 +1,22 @@ +/* contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql */ + +-- complain if script is sourced in psql, rather than via ALTER EXTENSION +\echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit + +-- +-- pg_get_wal_files() +-- +CREATE FUNCTION pg_get_wal_files( + IN start_lsn pg_lsn, + IN end_lsn pg_lsn DEFAULT NULL, + OUT wal_file text, + OUT segment_start_lsn pg_lsn, + OUT segment_end_lsn pg_lsn +) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'pg_get_wal_files' +LANGUAGE C PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO pg_read_server_files; diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index a172f9e2b40..85d76fe37c4 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -13,6 +13,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/timeline.h" #include "access/xlog.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" @@ -22,6 +23,7 @@ #include "funcapi.h" #include "miscadmin.h" #include "port/pg_bitutils.h" +#include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" @@ -38,6 +40,7 @@ PG_MODULE_MAGIC_EXT( ); PG_FUNCTION_INFO_V1(pg_get_wal_block_info); +PG_FUNCTION_INFO_V1(pg_get_wal_files); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -45,7 +48,7 @@ PG_FUNCTION_INFO_V1(pg_get_wal_stats); PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal); static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn); -static XLogRecPtr GetCurrentLSN(void); +static XLogRecPtr GetCurrentLSN(TimeLineID *current_tli); static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn); static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader); static void GetWALRecordInfo(XLogReaderState *record, Datum *values, @@ -68,11 +71,21 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Metadata for one retained segment in the current timeline's history. */ +typedef struct WalTimeSegment +{ + XLogSegNo segno; /* segment number, used for WAL ordering */ + TimeLineID tli; /* timeline containing this segment */ +} WalTimeSegment; + +static int wal_time_segment_cmp(const void *a, const void *b); +static WalTimeSegment *GetWalTimeSegments(int *nsegments); + /* - * Return the LSN up to which the server has WAL. + * Return the LSN up to which the server has WAL, and optionally its timeline. */ static XLogRecPtr -GetCurrentLSN(void) +GetCurrentLSN(TimeLineID *current_tli) { XLogRecPtr curr_lsn; @@ -81,9 +94,15 @@ GetCurrentLSN(void) * callback read_local_xlog_page_no_wait does. */ if (!RecoveryInProgress()) - curr_lsn = GetFlushRecPtr(NULL); + curr_lsn = GetFlushRecPtr(current_tli); else - curr_lsn = GetXLogReplayRecPtr(NULL); + { + curr_lsn = GetXLogReplayRecPtr(current_tli); + if (current_tli != NULL && *current_tli == 0) + { + *current_tli = GetWALInsertionTimeLineIfSet(); + } + } Assert(XLogRecPtrIsValid(curr_lsn)); @@ -190,6 +209,94 @@ ReadNextXLogRecord(XLogReaderState *xlogreader) return record; } +/* qsort comparator that puts segments in WAL order. */ +static int +wal_time_segment_cmp(const void *a, const void *b) +{ + const WalTimeSegment *seg1 = (const WalTimeSegment *) a; + const WalTimeSegment *seg2 = (const WalTimeSegment *) b; + + if (seg1->segno < seg2->segno) + return -1; + if (seg1->segno > seg2->segno) + return 1; + return 0; +} + +/* + * Get the WAL segments that form the history of the server's current + * timeline. Other timelines may have files with the same segment number in + * pg_wal, so choose the timeline that is valid at the end of each segment, + * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. + */ +static WalTimeSegment * +GetWalTimeSegments(int *nsegments) +{ + WalTimeSegment *segments = NULL; + DIR *dir; + struct dirent *de; + List *history; + XLogRecPtr current_lsn; + TimeLineID current_tli; + int allocated = 0; + int count = 0; + + current_lsn = GetCurrentLSN(&current_tli); + + history = readTimeLineHistory(current_tli); + dir = AllocateDir(XLOGDIR); + while ((de = ReadDir(dir, XLOGDIR)) != NULL) + { + TimeLineID file_tli; + XLogSegNo segno; + XLogRecPtr seg_start; + XLogRecPtr seg_end; + + if (!IsXLogFileName(de->d_name)) + continue; + + XLogFromFileName(de->d_name, &file_tli, &segno, wal_segment_size); + seg_start = segno * wal_segment_size; + seg_end = seg_start + wal_segment_size - 1; + + if (seg_start >= current_lsn || + file_tli != tliOfPointInHistory(seg_end, history)) + continue; + + if (count == allocated) + { + allocated = allocated ? allocated * 2 : 16; + if (segments == NULL) + segments = palloc_array(WalTimeSegment, allocated); + else + segments = repalloc_array(segments, WalTimeSegment, allocated); + } + + MemSet(&segments[count], 0, sizeof(WalTimeSegment)); + segments[count].segno = segno; + segments[count].tli = file_tli; + count++; + } + FreeDir(dir); + list_free_deep(history); + + if (count == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no retained WAL segments are available")); + + qsort(segments, count, sizeof(WalTimeSegment), wal_time_segment_cmp); + +#ifdef USE_ASSERT_CHECKING + /* A segment number identifies at most one file in this timeline history. */ + for (int i = 1; i < count; i++) + Assert(segments[i - 1].segno != segments[i].segno); +#endif + + *nsegments = count; + return segments; +} + /* * Output values that make up a row describing caller's WAL record. * @@ -483,7 +590,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) HeapTuple tuple; lsn = PG_GETARG_LSN(0); - curr_lsn = GetCurrentLSN(); + curr_lsn = GetCurrentLSN(NULL); if (lsn > curr_lsn) ereport(ERROR, @@ -516,6 +623,100 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * List the retained WAL segment files intersecting [start_lsn, end_lsn), or + * the segment containing start_lsn when end_lsn is omitted or equal to it. + * Report complete segment boundaries, not the range clipped to the inputs. + */ +Datum +pg_get_wal_files(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_FILES_COLS 3 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + XLogRecPtr current_lsn; + XLogSegNo first_segno; + XLogSegNo last_segno; + bool point_lookup; + WalTimeSegment *segments; + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + int nsegments; + int segment_index = 0; + + InitMaterializedSRF(fcinfo, 0); + + /* A NULL start produces an empty set. */ + if (PG_ARGISNULL(0)) + PG_RETURN_VOID(); + + start_lsn = PG_GETARG_LSN(0); + end_lsn = PG_ARGISNULL(1) ? start_lsn : PG_GETARG_LSN(1); + + if (start_lsn < XLOG_BLCKSZ) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not read WAL at LSN %X/%08X", + LSN_FORMAT_ARGS(start_lsn))); + + ValidateInputLSNs(start_lsn, &end_lsn); + point_lookup = start_lsn == end_lsn; + current_lsn = GetCurrentLSN(NULL); + + /* + * At a segment boundary, the current LSN can point just past the last + * retained WAL byte, before the next segment file has been created. In + * that case, return the preceding segment containing the available WAL. + */ + if (point_lookup && start_lsn == current_lsn && + XLogSegmentOffset(start_lsn, wal_segment_size) == 0) + XLByteToPrevSeg(start_lsn, first_segno, wal_segment_size); + else + XLByteToSeg(start_lsn, first_segno, wal_segment_size); + if (point_lookup) + last_segno = first_segno; + else + XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); + segments = GetWalTimeSegments(&nsegments); + + for (XLogSegNo segno = first_segno;; segno++) + { + Datum values[PG_GET_WAL_FILES_COLS]; + bool nulls[PG_GET_WAL_FILES_COLS] = {0}; + XLogRecPtr segment_start_lsn = segno * wal_segment_size; + XLogRecPtr segment_end_lsn = segment_start_lsn + wal_segment_size; + char fname[MAXFNAMELEN]; + + while (segment_index < nsegments && + segments[segment_index].segno < segno) + segment_index++; + + if (segment_index >= nsegments || + segments[segment_index].segno != segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the requested range is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) segno)); + + XLogFileName(fname, segments[segment_index].tli, segno, + wal_segment_size); + values[0] = CStringGetTextDatum(fname); + values[1] = LSNGetDatum(segment_start_lsn); + values[2] = LSNGetDatum(segment_end_lsn); + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, + values, nulls); + + if (segno == last_segno) + break; + + CHECK_FOR_INTERRUPTS(); + } + + pfree(segments); + PG_RETURN_VOID(); +#undef PG_GET_WAL_FILES_COLS +} + /* * Validate start and end LSNs coming from the function inputs. * @@ -525,7 +726,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn) { - XLogRecPtr curr_lsn = GetCurrentLSN(); + XLogRecPtr curr_lsn = GetCurrentLSN(NULL); if (start_lsn > curr_lsn) ereport(ERROR, @@ -831,7 +1032,7 @@ Datum pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); if (start_lsn > end_lsn) ereport(ERROR, @@ -849,7 +1050,7 @@ Datum pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); bool stats_per_record = PG_GETARG_BOOL(1); if (start_lsn > end_lsn) diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control index efa3cb2cfe5..5f574b865bb 100644 --- a/contrib/pg_walinspect/pg_walinspect.control +++ b/contrib/pg_walinspect/pg_walinspect.control @@ -1,5 +1,5 @@ # pg_walinspect extension comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log' -default_version = '1.1' +default_version = '1.2' module_pathname = '$libdir/pg_walinspect' relocatable = true diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql index e35c4f3ed15..b935a067b3e 100644 --- a/contrib/pg_walinspect/sql/oldextversions.sql +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -27,10 +27,11 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; --- List what version 1.1 contains. +-- List what the current version contains. \dx+ pg_walinspect SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 1e64a22d29a..3176a17d855 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -51,6 +51,30 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +SELECT * FROM pg_get_wal_files('0/0', '0/1'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -110,6 +134,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; @@ -122,6 +148,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE pg_read_server_files FROM regress_pg_walinspect; @@ -134,6 +162,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes @@ -143,6 +173,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; @@ -152,6 +184,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up diff --git a/contrib/pg_walinspect/t/001_timeline.pl b/contrib/pg_walinspect/t/001_timeline.pl new file mode 100644 index 00000000000..69cd53449e8 --- /dev/null +++ b/contrib/pg_walinspect/t/001_timeline.pl @@ -0,0 +1,73 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test that pg_get_wal_files() follows the current timeline's history across +# a promotion. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +sub test_timeline_history +{ + my ($suffix, $archive_mode) = @_; + my $primary = PostgreSQL::Test::Cluster->new("primary_$suffix"); + my $standby = PostgreSQL::Test::Cluster->new("standby_$suffix"); + + $primary->init(allows_streaming => 1); + $primary->append_conf('postgresql.conf', "autovacuum = off\n"); + if ($archive_mode) + { + $primary->append_conf( + 'postgresql.conf', + "archive_mode = on\narchive_command = 'true'\n"); + } + $primary->start; + $primary->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + my $boundary_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_lsn()'); + my $boundary_offset = $primary->safe_psql('postgres', + "SELECT file_offset FROM pg_walfile_name_offset('$boundary_lsn')"); + is($boundary_offset, '0', + "WAL is at a segment boundary with archive_mode=$archive_mode"); + my $boundary_files = $primary->safe_psql('postgres', + "SELECT count(*) FROM pg_get_wal_files('$boundary_lsn')"); + is($boundary_files, '1', + "point lookup works at a segment boundary with archive_mode=$archive_mode"); + + my $backup_name = "backup_$suffix"; + $primary->backup($backup_name); + $standby->init_from_backup($primary, $backup_name, has_streaming => 1); + $standby->start; + + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (1)'); + my $start_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (2)'); + $primary->wait_for_catchup($standby); + + $standby->promote; + $standby->safe_psql('postgres', 'INSERT INTO test_table VALUES (3)'); + + my $result = $standby->safe_psql( + 'postgres', qq{ +SELECT count(DISTINCT left(wal_file, 8)) = 2, + min(left(wal_file, 8)) = '00000001', + max(left(wal_file, 8)) = '00000002' +FROM pg_get_wal_files('$start_lsn', pg_current_wal_lsn())}); + is($result, 't|t|t', + "WAL files span the timeline switch with archive_mode=$archive_mode"); + + $standby->stop; + $primary->stop; + return; +} + +test_timeline_history('archive_off', 0); +test_timeline_history('archive_on', 1); + +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 79c3ead40bc..99dfe883b45 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -34,9 +34,8 @@ </note> <tip> <para> - All of the <filename>pg_walinspect</filename> functions that show - information about records that fall within a certain LSN range are - permissive about accepting <replaceable>end_lsn</replaceable> + All of the <filename>pg_walinspect</filename> functions that accept an LSN + range are permissive about accepting <replaceable>end_lsn</replaceable> arguments that are after the server's current LSN. Using an <replaceable>end_lsn</replaceable> <quote>from the future</quote> will not raise an error. @@ -60,6 +59,38 @@ <title>General Functions</title> <variablelist> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-files"> + <term> + <function>pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record</function> + </term> + + <listitem> + <para> + If <replaceable>end_lsn</replaceable> is specified and differs from + <replaceable>start_lsn</replaceable>, returns each retained WAL segment + file that intersects the half-open WAL range + [<replaceable>start_lsn</replaceable>, + <replaceable>end_lsn</replaceable>). If + <replaceable>end_lsn</replaceable> is omitted, is null, or equals + <replaceable>start_lsn</replaceable>, returns the segment containing + <replaceable>start_lsn</replaceable>. The + <structfield>wal_file</structfield> column contains the file name, while + <structfield>segment_start_lsn</structfield> and + <structfield>segment_end_lsn</structfield> contain the complete segment + boundaries rather than boundaries clipped to the requested range. + If a point lookup specifies the current WAL LSN and it lies exactly on + a segment boundary, the preceding segment is returned, since the next + segment might not have been created yet. + </para> + <para> + File names use the timeline valid for each segment in the current + timeline's history. This allows a returned range to cross a timeline + switch. The function reports an error if any required segment file is + not currently retained in <filename>pg_wal</filename>. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d001f56efae..3ab48bd75ca 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3430,6 +3430,7 @@ WalSummarizerData WalSummaryFile WalSummaryIO WalTimeSample +WalTimeSegment WalUsage WalWriteMethod WalWriteMethodOps -- 2.50.1 (Apple Git-155) =
v3-0002-pg_walinspect-add-function-to-locate-WAL-by-time.patch (application/octet-stream, 60.1K) download From 8f5353047c253f1db9f0e970bf25af25a4b3b45d Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v3 2/2] pg_walinspect: add function to locate WAL by time Add pg_get_wal_location_at_time(), which uses timestamped WAL records to locate a useful WAL range around a requested wall-clock time. Use an approximate binary search over retained segments and allow callers to widen the requested interval when timestamp reordering makes the initial result insufficient. Move WAL record timestamp extraction from recovery into xlogreader so pg_walinspect can recognize the same timestamp-bearing record types without duplicating that logic. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- .../pg_walinspect/expected/oldextversions.out | 9 +- .../pg_walinspect/expected/pg_walinspect.out | 164 ++++++ contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 21 + contrib/pg_walinspect/pg_walinspect.c | 485 +++++++++++++++++- contrib/pg_walinspect/sql/pg_walinspect.sql | 121 +++++ contrib/pg_walinspect/t/002_time_search.pl | 98 ++++ doc/src/sgml/pgwalinspect.sgml | 112 ++++ src/backend/access/transam/xlogreader.c | 36 ++ src/backend/access/transam/xlogrecovery.c | 48 +- src/include/access/xlogreader.h | 3 + src/tools/pgindent/typedefs.list | 1 + 12 files changed, 1050 insertions(+), 49 deletions(-) create mode 100644 contrib/pg_walinspect/t/002_time_search.pl diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 311017d66ca..6228ad9ef39 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -51,15 +51,16 @@ ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; -- List what the current version contains. \dx+ pg_walinspect - Objects in extension "pg_walinspect" - Object description -------------------------------------------------------- + Objects in extension "pg_walinspect" + Object description +---------------------------------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) function pg_get_wal_files(pg_lsn,pg_lsn) + function pg_get_wal_location_at_time(timestamp with time zone,interval,interval) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(5 rows) +(6 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index 34864c65ec7..341bb543295 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -139,6 +139,145 @@ FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + timestamped_ok +---------------- + t +(1 row) + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | end_ok +----------+-------- + t | t +(1 row) + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + start_ok | current_end_ok | end_lsn_ok | lsn_ok | records_ok +----------+----------------+------------+--------+------------ + t | t | t | t | t +(1 row) + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; + defaults_ok +------------- + t +(1 row) + +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +ERROR: before interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +ERROR: after interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +ERROR: could not locate a WAL time boundary +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); +ERROR: requested time follows the available WAL range +-- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset @@ -231,6 +370,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no has_function_privilege @@ -268,6 +414,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -285,6 +438,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -315,6 +470,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -330,6 +492,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 6f8b4a6359a..79e94e33e4d 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -41,6 +41,7 @@ tests += { 'tap': { 'tests': [ 't/001_timeline.pl', + 't/002_time_search.pl', ], }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql index 9cc9523b4dc..f72d99a3ceb 100644 --- a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -3,6 +3,27 @@ -- complain if script is sourced in psql, rather than via ALTER EXTENSION \echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit +-- +-- pg_get_wal_location_at_time() +-- +CREATE FUNCTION pg_get_wal_location_at_time( + IN target_time timestamptz, + IN before interval DEFAULT '1 minute', + IN after interval DEFAULT '1 minute', + OUT start_timestamp timestamptz, + OUT start_lsn pg_lsn, + OUT end_timestamp timestamptz, + OUT end_lsn pg_lsn +) +AS 'MODULE_PATHNAME', 'pg_get_wal_location_at_time' +LANGUAGE C STRICT PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) + TO pg_read_server_files; + -- -- pg_get_wal_files() -- diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index 85d76fe37c4..09329c17972 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -12,6 +12,8 @@ */ #include "postgres.h" +#include <sys/stat.h> + #include "access/htup_details.h" #include "access/timeline.h" #include "access/xlog.h" @@ -26,7 +28,9 @@ #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgrprotos.h" #include "utils/pg_lsn.h" +#include "utils/timestamp.h" #include "utils/tuplestore.h" /* @@ -41,6 +45,7 @@ PG_MODULE_MAGIC_EXT( PG_FUNCTION_INFO_V1(pg_get_wal_block_info); PG_FUNCTION_INFO_V1(pg_get_wal_files); +PG_FUNCTION_INFO_V1(pg_get_wal_location_at_time); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -71,15 +76,52 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Selected WAL position and timestamp for one time boundary. */ +typedef struct WalTimeBoundary +{ + bool found; /* matching record was found */ + TimestampTz time; /* record's local event time */ + XLogRecPtr lsn; /* record's start LSN */ + int segment_index; /* index of the segment containing the record */ +} WalTimeBoundary; + /* Metadata for one retained segment in the current timeline's history. */ typedef struct WalTimeSegment { XLogSegNo segno; /* segment number, used for WAL ordering */ TimeLineID tli; /* timeline containing this segment */ + bool scanned; /* segment has already been decoded */ + bool has_timestamp; /* segment contains timestamped records */ + TimestampTz min_time; /* minimum record timestamp in segment */ + TimestampTz max_time; /* maximum record timestamp in segment */ + WalTimeBoundary lower; /* best lower anchor in segment */ + WalTimeBoundary upper; /* best upper anchor in segment */ } WalTimeSegment; static int wal_time_segment_cmp(const void *a, const void *b); -static WalTimeSegment *GetWalTimeSegments(int *nsegments); +static WalTimeSegment *GetWalTimeSegments(int *nsegments, + XLogRecPtr *current_lsn_out); +static int FindWalTimeCandidate(WalTimeSegment *segments, + int *search_segments, + int *nsearch_segments, + TimestampTz target_time, + bool search_left, + TimestampTz wanted_start, + TimestampTz wanted_end, + int *candidate_position, + int *nscanned); +static void ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, + TimestampTz wanted_end); +static void MergeWalTimeSegment(WalTimeSegment *segment, + WalTimeBoundary *lower, + WalTimeBoundary *upper); +static void ProbeLowerBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); +static void ProbeUpperBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); /* * Return the LSN up to which the server has WAL, and optionally its timeline. @@ -223,6 +265,46 @@ wal_time_segment_cmp(const void *a, const void *b) return 0; } +/* + * Probe a timestamped record at or before a requested time boundary. Among + * records examined so far, prefer the latest time, breaking ties in favor + * of the earlier WAL record. The caller only passes qualifying records, so + * a greater timestamp is closer to the requested boundary. + */ +static void +ProbeLowerBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time > boundary->time || + (time == boundary->time && lsn < boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + +/* + * Probe a timestamped record at or after a requested time boundary. Among + * records examined so far, prefer the earliest time, breaking ties in favor of + * the later WAL record. The caller only passes qualifying records, so a + * smaller timestamp is closer to the requested boundary. + */ +static void +ProbeUpperBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time < boundary->time || + (time == boundary->time && lsn > boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + /* * Get the WAL segments that form the history of the server's current * timeline. Other timelines may have files with the same segment number in @@ -230,7 +312,7 @@ wal_time_segment_cmp(const void *a, const void *b) * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. */ static WalTimeSegment * -GetWalTimeSegments(int *nsegments) +GetWalTimeSegments(int *nsegments, XLogRecPtr *current_lsn_out) { WalTimeSegment *segments = NULL; DIR *dir; @@ -242,6 +324,8 @@ GetWalTimeSegments(int *nsegments) int count = 0; current_lsn = GetCurrentLSN(&current_tli); + if (current_lsn_out != NULL) + *current_lsn_out = current_lsn; history = readTimeLineHistory(current_tli); dir = AllocateDir(XLOGDIR); @@ -297,6 +381,140 @@ GetWalTimeSegments(int *nsegments) return segments; } +/* + * Approximately locate the segment containing target_time by binary search. + * Record timestamps are expected to generally increase with WAL position. + * Remove segments without timestamped records from the search candidates, + * since they provide no value with which to choose either half. + */ +static int +FindWalTimeCandidate(WalTimeSegment *segments, int *search_segments, + int *nsearch_segments, + TimestampTz target_time, bool search_left, + TimestampTz wanted_start, TimestampTz wanted_end, + int *candidate_position, int *nscanned) +{ + int left = 0; + int right = *nsearch_segments - 1; + int result; + + while (left <= right) + { + int middle = left + (right - left) / 2; + int segment_index = search_segments[middle]; + WalTimeSegment *segment = &segments[segment_index]; + + if (!segment->scanned) + { + ScanWalTimeSegment(segment, segment_index, wanted_start, wanted_end); + (*nscanned)++; + } + + if (!segment->has_timestamp) + { + memmove(&search_segments[middle], &search_segments[middle + 1], + (*nsearch_segments - middle - 1) * sizeof(int)); + (*nsearch_segments)--; + right--; + continue; + } + else if (target_time < segment->min_time) + right = middle - 1; + else if (target_time > segment->max_time) + left = middle + 1; + else + { + if (candidate_position != NULL) + *candidate_position = middle; + return search_segments[middle]; + } + } + + if (*nsearch_segments == 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("No retained WAL segment contains a timestamped record.")); + + if (search_left) + result = Max(right, 0); + else + result = Min(left, *nsearch_segments - 1); + + if (candidate_position != NULL) + *candidate_position = result; + return search_segments[result]; +} + +/* + * Inspect timestamped records beginning in one segment and save its time + * range and best boundary candidates. XLogReadRecord() may read the following + * segment to assemble a record crossing the boundary, but that record is + * attributed to the segment containing its starting LSN. + */ +static void +ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, TimestampTz wanted_end) +{ + XLogRecPtr seg_start = segment->segno * wal_segment_size; + XLogRecPtr seg_end = seg_start + wal_segment_size; + XLogReaderState *xlogreader; + + Assert(!segment->scanned); + segment->scanned = true; + xlogreader = InitXLogReaderState(seg_start); + + while (ReadNextXLogRecord(xlogreader)) + { + TimestampTz record_time; + + if (xlogreader->ReadRecPtr >= seg_end) + break; + + if (!GetXLogRecordTimestamp(xlogreader, &record_time)) + continue; + + if (!segment->has_timestamp) + { + segment->has_timestamp = true; + segment->min_time = segment->max_time = record_time; + } + else + { + segment->min_time = Min(segment->min_time, record_time); + segment->max_time = Max(segment->max_time, record_time); + } + + if (record_time <= wanted_start) + ProbeLowerBoundary(&segment->lower, record_time, + xlogreader->ReadRecPtr, segment_index); + + else if (record_time >= wanted_end) + ProbeUpperBoundary(&segment->upper, record_time, + xlogreader->ReadRecPtr, segment_index); + + CHECK_FOR_INTERRUPTS(); + } + + pfree(xlogreader->private_data); + XLogReaderFree(xlogreader); +} + +/* Merge one segment's boundary candidates into the result. */ +static void +MergeWalTimeSegment(WalTimeSegment *segment, WalTimeBoundary *lower, + WalTimeBoundary *upper) +{ + Assert(segment->scanned); + + if (lower != NULL && segment->lower.found) + ProbeLowerBoundary(lower, segment->lower.time, segment->lower.lsn, + segment->lower.segment_index); + if (upper != NULL && segment->upper.found) + ProbeUpperBoundary(upper, segment->upper.time, segment->upper.lsn, + segment->upper.segment_index); +} + /* * Output values that make up a row describing caller's WAL record. * @@ -623,6 +841,267 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * Locate a WAL region around a wall-clock time range using timestamped WAL + * records. + * + * Locate each boundary by an approximate binary search over retained WAL + * segments, assuming that record timestamps generally increase with WAL + * position. Also examine one preceding segment for the lower boundary and + * one following segment for the upper boundary to allow for limited timestamp + * reordering. The lower boundary uses the latest timestamped record at or + * before the requested start time, while the upper boundary uses the earliest + * one at or after the requested end time. A future upper bound is first + * capped at the current time. If no upper timestamped record is available, + * use the current WAL position. On success, the boundaries form a forward + * WAL range, but are not necessarily the globally closest timestamped records + * to the requested bounds. + */ +Datum +pg_get_wal_location_at_time(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_LOCATION_AT_TIME_COLS 4 + TimestampTz target_time = PG_GETARG_TIMESTAMPTZ(0); + Interval *before = PG_GETARG_INTERVAL_P(1); + Interval *after = PG_GETARG_INTERVAL_P(2); + Interval zero = {0}; + Interval one_day = {.day = 1}; + TimestampTz wanted_start; + TimestampTz wanted_end; + TimestampTz current_time; + XLogRecPtr current_lsn; + XLogSegNo current_segno; + WalTimeSegment *segments; + WalTimeBoundary lower = {0}; + WalTimeBoundary upper = {0}; + Datum values[PG_GET_WAL_LOCATION_AT_TIME_COLS]; + bool nulls[PG_GET_WAL_LOCATION_AT_TIME_COLS] = {0}; + TupleDesc tupdesc; + HeapTuple tuple; + int nsegments; + int *search_segments; + int nsearch_segments; + int lower_candidate; + int upper_candidate; + int lower_candidate_position; + int nupper_search_segments; + int lower_start; + int upper_end; + int upper_segment_index; + int nscanned = 0; + bool upper_bound_capped = false; + char lower_fname[MAXFNAMELEN]; + char upper_fname[MAXFNAMELEN]; + + if (TIMESTAMP_NOT_FINITE(target_time)) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("target time must be finite")); + if (INTERVAL_NOT_FINITE(before) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must not exceed one day")); + if (INTERVAL_NOT_FINITE(after) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must not exceed one day")); + + /* The timestamp functions provide the normal overflow checks. */ + wanted_start = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_mi_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(before))); + wanted_end = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_pl_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(after))); + current_time = GetCurrentTimestamp(); + + /* Cap a future upper bound at the time currently available. */ + if (wanted_end > current_time) + { + if (wanted_start >= current_time) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("The requested lower bound is not earlier than the current time.")); + + wanted_end = current_time; + upper_bound_capped = true; + } + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + segments = GetWalTimeSegments(&nsegments, &current_lsn); + XLByteToPrevSeg(current_lsn, current_segno, wal_segment_size); + + /* + * A capped upper bound may need the current WAL position as its endpoint. + * Detect a missing WAL tail before decoding any candidate segments. + */ + if (upper_bound_capped && + segments[nsegments - 1].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The current WAL LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(current_lsn), + (uint64) current_segno)); + + search_segments = palloc_array(int, nsegments); + for (int i = 0; i < nsegments; i++) + search_segments[i] = i; + nsearch_segments = nsegments; + lower_candidate = FindWalTimeCandidate(segments, search_segments, + &nsearch_segments, wanted_start, + true, wanted_start, wanted_end, + &lower_candidate_position, + &nscanned); + + /* Search for the later upper boundary from the lower candidate onward. */ + nupper_search_segments = nsearch_segments - lower_candidate_position; + upper_candidate = FindWalTimeCandidate(segments, + search_segments + lower_candidate_position, + &nupper_search_segments, wanted_end, + false, wanted_start, wanted_end, + NULL, + &nscanned); + XLogFileName(lower_fname, segments[lower_candidate].tli, + segments[lower_candidate].segno, wal_segment_size); + XLogFileName(upper_fname, segments[upper_candidate].tli, + segments[upper_candidate].segno, wal_segment_size); + ereport(DEBUG1, + errmsg_internal("WAL time search selected lower segment %s and upper segment %s from %d retained segments", + lower_fname, upper_fname, + nsegments)); + + /* + * Allow for limited timestamp reordering by examining one preceding + * segment for the lower boundary and one following segment for the upper + * boundary. Do not extend across a missing segment. + */ + lower_start = lower_candidate; + if (lower_start > 0 && + segments[lower_start - 1].segno + 1 == segments[lower_start].segno) + lower_start--; + upper_end = upper_candidate; + if (upper_end + 1 < nsegments && + segments[upper_end].segno + 1 == segments[upper_end + 1].segno) + upper_end++; + + for (int i = lower_start; i <= lower_candidate; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], &lower, NULL); + } + for (int i = upper_candidate; i <= upper_end; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], NULL, &upper); + } + + ereport(DEBUG1, + errmsg_internal("WAL time search decoded %d of %d retained segments", + nscanned, nsegments)); + + if (!lower.found) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("The WAL search did not find a timestamped record at or before the requested lower bound.")); + + /* + * There may be no timestamped record after the requested upper bound on a + * quiet system. In that case, use the current WAL position so that the + * returned range still includes all WAL currently available. + */ + if (!upper.found) + { + upper.lsn = current_lsn; + + /* + * The last enumerated segment should contain current_lsn. Verify + * this because pg_wal can change while it is being enumerated; + * otherwise the continuity check below would not cover a missing WAL + * tail. + */ + upper_segment_index = nsegments - 1; + if (segments[upper_segment_index].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) current_segno)); + } + else + upper_segment_index = upper.segment_index; + + if (lower.lsn >= upper.lsn) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("could not find a valid WAL range for the requested time window"), + errdetail("The matching lower boundary does not precede the upper boundary in WAL.")); + + Assert(lower.segment_index >= 0 && lower.segment_index < nsegments); + Assert(upper_segment_index >= 0 && upper_segment_index < nsegments); + for (int i = lower.segment_index + 1; i <= upper_segment_index; i++) + { + /* + * The returned LSN range must be continuously readable from pg_wal; + * otherwise, functions such as pg_get_wal_records_info() would fail + * when scanning the range across a missing segment. + */ + if (segments[i - 1].segno + 1 != segments[i].segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) (segments[i - 1].segno + 1))); + } + + values[0] = TimestampTzGetDatum(lower.time); + values[1] = LSNGetDatum(lower.lsn); + if (upper.found) + values[2] = TimestampTzGetDatum(upper.time); + else + nulls[2] = true; + values[3] = LSNGetDatum(upper.lsn); + + pfree(segments); + tuple = heap_form_tuple(tupdesc, values, nulls); + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +#undef PG_GET_WAL_LOCATION_AT_TIME_COLS +} + /* * List the retained WAL segment files intersecting [start_lsn, end_lsn), or * the segment containing start_lsn when end_lsn is omitted or equal to it. @@ -676,7 +1155,7 @@ pg_get_wal_files(PG_FUNCTION_ARGS) last_segno = first_segno; else XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); - segments = GetWalTimeSegments(&nsegments); + segments = GetWalTimeSegments(&nsegments, NULL); for (XLogSegNo segno = first_segno;; segno++) { diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 3176a17d855..4b37a25fa41 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -75,6 +75,117 @@ SELECT count(*) = 1 AS future_end_ok FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== + +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); + +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); + +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); + +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -134,6 +245,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no @@ -148,6 +261,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -162,6 +277,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; @@ -173,6 +290,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -184,6 +303,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; diff --git a/contrib/pg_walinspect/t/002_time_search.pl b/contrib/pg_walinspect/t/002_time_search.pl new file mode 100644 index 00000000000..96a5c06eb76 --- /dev/null +++ b/contrib/pg_walinspect/t/002_time_search.pl @@ -0,0 +1,98 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test the bounded binary search for WAL time boundaries across segments. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $node = PostgreSQL::Test::Cluster->new('time_search'); +$node->init(extra => ['--wal-segsize=1']); +$node->append_conf('postgresql.conf', "autovacuum = off\n"); +$node->start; +$node->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); +$node->safe_psql('postgres', 'SELECT pg_switch_wal()'); + +my ($lower_time, $upper_time); +for my $i (0 .. 8) +{ + $node->safe_psql('postgres', "INSERT INTO test_table VALUES ($i)"); + $lower_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 3; + $upper_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 5; + $node->safe_psql('postgres', 'SELECT pg_switch_wal()') if $i < 8; +} + +my $log_offset = -s $node->logfile; +my $result = $node->safe_psql( + 'postgres', qq{ +SET log_min_messages = debug1; +SELECT start_timestamp <= '$lower_time'::timestamptz - interval '1 microsecond', + end_timestamp >= '$upper_time'::timestamptz, + start_lsn < end_lsn, + pg_walfile_name(start_lsn) <> pg_walfile_name(end_lsn) +FROM pg_get_wal_location_at_time('$lower_time', + interval '1 microsecond', + '$upper_time'::timestamptz - + '$lower_time'::timestamptz)}); +is($result, 't|t|t|t', 'binary search finds outer boundaries across WAL segments'); + +my $log = slurp_file($node->logfile, $log_offset); +$log =~ /WAL time search decoded (\d+) of (\d+) retained segments/ + or die "could not find WAL time search statistics in server log"; +cmp_ok($1, '<', $2, 'binary search does not decode all retained WAL'); + +# A window containing no timestamped record is enclosed by the timestamped +# records immediately outside it. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS empty_window_start \gset +SELECT count(*) AS ignored FROM generate_series(1, 100000) \gset +SELECT clock_timestamp() AS empty_window_end \gset +INSERT INTO test_table VALUES (9); + +SELECT location.start_timestamp < :'empty_window_start'::timestamptz, + location.end_timestamp > :'empty_window_end'::timestamptz, + location.start_lsn < location.end_lsn +FROM pg_get_wal_location_at_time( + :'empty_window_start', + interval '1 microsecond', + :'empty_window_end'::timestamptz - + :'empty_window_start'::timestamptz) AS location;}); +is($result, 't|t|t', + 'outer anchors enclose a window with no timestamped record'); + +# With no later timestamped record, use the current WAL position as the upper +# boundary. The returned range must include WAL generated before the only +# commit record in the requested window. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS window_start \gset +BEGIN; +SELECT pg_current_wal_insert_lsn() AS delete_lsn \gset +DELETE FROM test_table WHERE a = 5; +COMMIT; +SELECT clock_timestamp() AS window_end \gset + +SELECT location.start_timestamp <= + :'window_start'::timestamptz - interval '1 microsecond', + location.end_timestamp IS NULL, + location.start_lsn < :'delete_lsn'::pg_lsn, + location.end_lsn > :'delete_lsn'::pg_lsn, + EXISTS (SELECT + FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'DELETE') +FROM pg_get_wal_location_at_time( + :'window_start', + interval '1 microsecond', + :'window_end'::timestamptz - :'window_start'::timestamptz) AS location;}); +is($result, 't|t|t|t|t', + 'current WAL fallback includes changes preceding the only commit in the window'); + +$node->stop; +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 99dfe883b45..ab21de4501a 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -91,6 +91,118 @@ </listitem> </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-location-at-time"> + <term> + <function>pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute') returns record</function> + </term> + + <listitem> + <para> + Locates a nearby WAL region using records containing local event + timestamps as historical time anchors. The requested time range extends + from <replaceable>target_time</replaceable> minus + <replaceable>before</replaceable> through + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable>. It returns + <structfield>start_timestamp</structfield>, + <structfield>start_lsn</structfield>, + <structfield>end_timestamp</structfield>, and + <structfield>end_lsn</structfield>. The effective upper bound is the + earlier of <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> and the current time. On successful + return, the following relationships hold: +<programlisting> +start_timestamp &lt;= target_time - before +end_timestamp &gt;= effective upper bound, or is null +start_lsn &lt; end_lsn +</programlisting> + The start boundary is the latest timestamped record found at or before + the requested start time, and the end boundary is the earliest + timestamped record found at or after the requested end time. If + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> is later than the current time, the + requested upper bound is first capped at the current time. If no + timestamped record is available at or after the upper bound, + <structfield>end_lsn</structfield> is the current flush or replay + position and <structfield>end_timestamp</structfield> is null. Both + interval arguments must be greater than zero and not greater than one + day. + </para> + <para> + Most WAL records do not contain a wall-clock timestamp. Historical + anchors can be transaction <literal>COMMIT</literal>, + <literal>COMMIT PREPARED</literal>, <literal>ABORT</literal>, + or <literal>ABORT PREPARED</literal> records, as well as restore point + records. These are the WAL record types from which PostgreSQL recovery + can extract timestamps. Their timestamps describe the corresponding + event; they do not assign timestamps to heap, index, or other records. + </para> + <para> + In particular, a commit timestamp describes when a transaction + committed, not when its earlier changes generated WAL. For example, a + transaction might update a row at 14:00 and commit at 15:00; the heap + WAL was generated around 14:00, while the commit record is timestamped + 15:00. Using timestamped records outside the requested time window as + boundaries helps include the WAL generated by transactions that commit + within the window. However, the returned LSNs do not identify the exact + generation time of heap, index, or other WAL records, and a transaction + can have generated WAL before the selected start boundary. On a cluster + with little timestamped activity, the returned timestamps can be far + from the requested bounds. + </para> + <para> + The function assumes that WAL record timestamps generally increase with + WAL position and uses an approximate binary search to locate each time + boundary. WAL record timestamps are not guaranteed to follow WAL order, + for example because of concurrent activity, clock adjustments, or + delayed WAL insertion. Such timestamp reordering can cause relevant + records to fall outside the returned range. Therefore, the returned + records are not guaranteed to be the globally closest timestamped + records to either requested time bound. If the returned range does not + contain the records of interest, the user can widen the search range and + repeat the search. If the matching lower boundary follows the upper + boundary in WAL, the function reports an error rather than returning a + range with the LSNs reversed. + </para> + <para> + Only WAL that is currently retained in <filename>pg_wal</filename> can + be searched. Unlike the other <application>pg_walinspect</application> + functions, this function may inspect retained WAL from the current + timeline's history as well as the current timeline. The function + reports an error when a lower boundary cannot be found by the bounded + search or when the resulting range would cross a missing WAL segment. + When an upper timestamped boundary cannot be found, the current WAL + position is returned instead, as described above. + </para> + <para> + For example, a DBA can locate WAL around activity near a given time and + then inspect the returned range: +<programlisting> +SELECT * +FROM pg_get_wal_location_at_time( + '2026-08-01 14:35:00+08', + before =&gt; '10 minutes', + after =&gt; '10 minutes'); + +SELECT * +FROM pg_get_wal_files('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); + +SELECT * +FROM pg_get_wal_records_info('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); +</programlisting> + The corresponding retained WAL files can be obtained with + <function>pg_get_wal_files</function>. To widen the search range, + increase <replaceable>before</replaceable> if the relevant activity + might have occurred earlier, or increase + <replaceable>after</replaceable> if it might have occurred later, and + repeat the search. Failure to find a record in the first returned range + does not necessarily mean that the record is absent from retained WAL. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 7db7c273b0c..02f49f64b3b 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -26,6 +26,7 @@ #endif #include "access/transam.h" +#include "access/xact.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" #include "access/xlogrecord.h" @@ -65,6 +66,41 @@ static void WALOpenSegmentInit(WALOpenSegment *seg, WALSegmentContext *segcxt, */ #define DEFAULT_DECODE_BUFFER_SIZE (64 * 1024) +/* + * Extract timestamp from WAL record. + * + * If the record contains a timestamp, returns true, and saves the timestamp + * in *recordXtime. If the record type has no timestamp, returns false. + * Currently, only transaction commit/abort records and restore points contain + * timestamps. + */ +bool +GetXLogRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) +{ + uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; + uint8 xact_info = info & XLOG_XACT_OPMASK; + uint8 rmid = XLogRecGetRmid(record); + + if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) + { + *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || + xact_info == XLOG_XACT_COMMIT_PREPARED)) + { + *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || + xact_info == XLOG_XACT_ABORT_PREPARED)) + { + *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; + return true; + } + return false; +} + /* * Construct a string in state->errormsg_buf explaining what's wrong with * the current record being read. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 54aaec9529f..06a5b10d092 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -356,7 +356,6 @@ static void xlog_outrec(StringInfo buf, XLogReaderState *record); static void xlog_block_info(StringInfo buf, XLogReaderState *record); static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, TimeLineID replayTLI); -static bool getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime); static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); @@ -2405,41 +2404,6 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, } -/* - * Extract timestamp from WAL record. - * - * If the record contains a timestamp, returns true, and saves the timestamp - * in *recordXtime. If the record type has no timestamp, returns false. - * Currently, only transaction commit/abort records and restore points contain - * timestamps. - */ -static bool -getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) -{ - uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; - uint8 xact_info = info & XLOG_XACT_OPMASK; - uint8 rmid = XLogRecGetRmid(record); - - if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) - { - *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || - xact_info == XLOG_XACT_COMMIT_PREPARED)) - { - *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || - xact_info == XLOG_XACT_ABORT_PREPARED)) - { - *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; - return true; - } - return false; -} - /* * Checks whether the current buffer page and backup page stored in the * WAL record are consistent or not. Before comparing the two pages, a @@ -2663,10 +2627,10 @@ recoveryStopsBefore(XLogReaderState *record) /* * Note: we must fetch recordXtime regardless of recoveryTarget setting. - * We don't expect getRecordTimestamp ever to fail, since we already know - * this is a commit or abort record; but test its result anyway. + * We don't expect GetXLogRecordTimestamp ever to fail, since we already + * know this is a commit or abort record; but test its result anyway. */ - if (getRecordTimestamp(record, &recordXtime) && + if (GetXLogRecordTimestamp(record, &recordXtime) && recoveryTarget == RECOVERY_TARGET_TIME) { /* @@ -2747,7 +2711,7 @@ recoveryStopsAfter(XLogReaderState *record) recoveryStopAfter = true; recoveryStopXid = InvalidTransactionId; recoveryStopLSN = InvalidXLogRecPtr; - (void) getRecordTimestamp(record, &recoveryStopTime); + (void) GetXLogRecordTimestamp(record, &recoveryStopTime); strlcpy(recoveryStopName, recordRestorePointData->rp_name, MAXFNAMELEN); ereport(LOG, @@ -2787,7 +2751,7 @@ recoveryStopsAfter(XLogReaderState *record) TransactionId recordXid; /* Update the last applied transaction timestamp */ - if (getRecordTimestamp(record, &recordXtime)) + if (GetXLogRecordTimestamp(record, &recordXtime)) SetLatestXTime(recordXtime); /* Extract the XID of the committed/aborted transaction */ @@ -3006,7 +2970,7 @@ recoveryApplyDelay(XLogReaderState *record) xact_info != XLOG_XACT_COMMIT_PREPARED) return false; - if (!getRecordTimestamp(record, &xtime)) + if (!GetXLogRecordTimestamp(record, &xtime)) return false; delayUntil = TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index 4a9a687e879..3016ae4d5da 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -39,6 +39,7 @@ #endif #include "access/xlogrecord.h" +#include "datatype/timestamp.h" #include "storage/buf.h" /* WALOpenSegment represents a WAL segment being read. */ @@ -440,5 +441,7 @@ extern bool XLogRecGetBlockTagExtended(XLogReaderState *record, uint8 block_id, RelFileLocator *rlocator, ForkNumber *forknum, BlockNumber *blknum, Buffer *prefetch_buffer); +extern bool GetXLogRecordTimestamp(XLogReaderState *record, + TimestampTz *recordXtime); #endif /* XLOGREADER_H */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3ab48bd75ca..fb81d5e862a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3429,6 +3429,7 @@ WalSndState WalSummarizerData WalSummaryFile WalSummaryIO +WalTimeBoundary WalTimeSample WalTimeSegment WalUsage -- 2.50.1 (Apple Git-155) =
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-08 22:36  surya poondla <suryapoondla4@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: surya poondla @ 2026-10-08 22:36 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi Chao,

Thanks for v3. Moving to outer anchors with the fallback to the current WAL
position addresses my concern with v2, and I appreciate the new tests
covering both the empty-window case and the DELETE scenario.

v3 applies cleanly to master (f884f359f5a) and builds without warnings.
Locally, the regression tests and both TAP tests pass, and so does my
earlier reproducer. The returned range now brackets the DELETE:

   start_lsn  |  end_lsn   | delete_lsn
  ------------+------------+------------
   0/017F0220 | 0/017F02E0 | 0/017F0280


However, CFBot's macOS - Meson job fails on v3 (
https://github.com/postgres/postgres-cfbot/actions/runs/37738280802/job/113183259446)
in 001_timeline.pl, during the archive_mode=0 pass:

  ERROR:  WAL segment needed for the requested range is missing
  DETAIL:  Segment 4 is not present in pg_wal.
  ... FROM pg_get_wal_files('0/04001790', pg_current_wal_lsn())

The standby's server log from that run points to a race in the test rather
than a bug in pg_get_wal_files():

  06:52:52.657  received promote request
  06:52:52.701  checkpoint starting: force
  06:52:52.938  checkpoint complete: ... 0 WAL file(s) added, 0 removed, 2
recycled
  06:52:52.972  ERROR:  WAL segment needed for the requested range is
missing

The checkpoint forced by the promotion recycled two WAL segments, and since
segment 4, which holds start_lsn, was gone 34 ms later, it must have been
one of them. On a faster machine the query runs before that checkpoint
completes, which would explain why the test passes locally.

Keeping that WAL on the standby makes the test deterministic. I confirmed
this locally with a copy of the test that forces a CHECKPOINT right after
promote(): it fails every time with the same "Segment 4 is not present"
error, and passes once the standby has wal_keep_size set:

$standby->init_from_backup($primary, $backup_name, has_streaming => 1);
$standby->append_conf('postgresql.conf', "wal_keep_size = '64MB'");

src/test/recovery/t/004_timeline_switch.pl uses wal_keep_size the same way.

One small question on 0001. For a point lookup at the current LSN on a
segment boundary, v3 now returns the preceding segment, which makes sense
given that the next file may not exist yet. pg_walfile_name(), though, maps
the same LSN to the next segment (it uses XLByteToSeg). So at that one LSN
the two functions name different files. I'm fine with the choice, since the
docs explain it, but it may be worth a sentence noting the difference from
pg_walfile_name().

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-09 01:28  Chao Li <li.evan.chao@gmail.com>
  parent: surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Chao Li @ 2026-10-09 01:28 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>



> On Oct 9, 2026, at 06:36, surya poondla <suryapoondla4@gmail.com> wrote:
> 
> Hi Chao,
> 
> Thanks for v3. Moving to outer anchors with the fallback to the current WAL position addresses my concern with v2, and I appreciate the new tests
> covering both the empty-window case and the DELETE scenario.
> 
> v3 applies cleanly to master (f884f359f5a) and builds without warnings.
> Locally, the regression tests and both TAP tests pass, and so does my earlier reproducer. The returned range now brackets the DELETE:
> 
>    start_lsn  |  end_lsn   | delete_lsn
>   ------------+------------+------------
>    0/017F0220 | 0/017F02E0 | 0/017F0280
> 
> 
> However, CFBot's macOS - Meson job fails on v3 (https://github.com/postgres/postgres-cfbot/actions/runs/37738280802/job/113183259446) in 001_timeline.pl, during the archive_mode=0 pass:
> 
>   ERROR:  WAL segment needed for the requested range is missing
>   DETAIL:  Segment 4 is not present in pg_wal.
>   ... FROM pg_get_wal_files('0/04001790', pg_current_wal_lsn())
> 
> The standby's server log from that run points to a race in the test rather than a bug in pg_get_wal_files():
> 
>   06:52:52.657  received promote request
>   06:52:52.701  checkpoint starting: force
>   06:52:52.938  checkpoint complete: ... 0 WAL file(s) added, 0 removed, 2 recycled
>   06:52:52.972  ERROR:  WAL segment needed for the requested range is missing
> 
> The checkpoint forced by the promotion recycled two WAL segments, and since segment 4, which holds start_lsn, was gone 34 ms later, it must have been
> one of them. On a faster machine the query runs before that checkpoint completes, which would explain why the test passes locally.
> 
> Keeping that WAL on the standby makes the test deterministic. I confirmed this locally with a copy of the test that forces a CHECKPOINT right after
> promote(): it fails every time with the same "Segment 4 is not present" error, and passes once the standby has wal_keep_size set:
> 
> $standby->init_from_backup($primary, $backup_name, has_streaming => 1);
> $standby->append_conf('postgresql.conf', "wal_keep_size = '64MB'");
> 
> src/test/recovery/t/004_timeline_switch.pl uses wal_keep_size the same way.

Fixed by setting wal_keep_size, which prevents the required WAL file from being recycled.

> 
> One small question on 0001. For a point lookup at the current LSN on a segment boundary, v3 now returns the preceding segment, which makes sense
> given that the next file may not exist yet. pg_walfile_name(), though, maps the same LSN to the next segment (it uses XLByteToSeg). So at that one LSN
> the two functions name different files. I'm fine with the choice, since the docs explain it, but it may be worth a sentence noting the difference from
> pg_walfile_name().
> 

I think the two functions do different things, although their names sound similar. pg_walfile_name(lsn) performs a pure calculation that maps an LSN to a WAL file name without checking if the file exists, while pg_get_wal_files(start_lsn, end_lsn) returns a list of WAL files found on disk. We can add a sentence to the doc to clarify this difference.

PFA v4:
* 0001 - Fixed the fragile TAP test, and enhanced the doc.
* 0002 - Tiny doc formatting changes.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments: download all (2 files, .tar.gz)

v4-0001-pg_walinspect-add-function-to-list-WAL-files-by-L.patch (application/octet-stream, 29.3K) download From dbe04551fc71969524c421ebb41bdc28ef6324ff Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v4 1/2] pg_walinspect: add function to list WAL files by LSN Add pg_get_wal_files(), which returns the retained WAL segment files intersecting an LSN range, together with each segment's complete LSN boundaries. A point lookup can be requested by omitting end_lsn or making it equal to start_lsn. Choose segment file names from the server's current timeline history so that ranges spanning a timeline switch are handled correctly. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- contrib/pg_walinspect/Makefile | 4 +- .../pg_walinspect/expected/oldextversions.out | 8 +- .../pg_walinspect/expected/pg_walinspect.out | 73 ++++++ contrib/pg_walinspect/meson.build | 6 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 22 ++ contrib/pg_walinspect/pg_walinspect.c | 219 +++++++++++++++++- contrib/pg_walinspect/pg_walinspect.control | 2 +- contrib/pg_walinspect/sql/oldextversions.sql | 5 +- contrib/pg_walinspect/sql/pg_walinspect.sql | 34 +++ contrib/pg_walinspect/t/001_timeline.pl | 74 ++++++ doc/src/sgml/pgwalinspect.sgml | 41 +++- src/tools/pgindent/typedefs.list | 1 + 12 files changed, 470 insertions(+), 19 deletions(-) create mode 100644 contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql create mode 100644 contrib/pg_walinspect/t/001_timeline.pl diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 22090f7716c..b7651231077 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -7,9 +7,11 @@ OBJS = \ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log" EXTENSION = pg_walinspect -DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql +DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql \ + pg_walinspect--1.1--1.2.sql REGRESS = pg_walinspect oldextversions +TAP_TESTS = 1 REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_walinspect/walinspect.conf diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 89953af3d0e..311017d66ca 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -46,18 +46,20 @@ SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; --- List what version 1.1 contains. +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; +-- List what the current version contains. \dx+ pg_walinspect Objects in extension "pg_walinspect" Object description ------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) + function pg_get_wal_files(pg_lsn,pg_lsn) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(4 rows) +(5 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index f955ff5d3c5..34864c65ec7 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -90,6 +90,54 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); t (1 row) +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + file_ok | start_ok | end_ok +---------+----------+-------- + t | t | t +(1 row) + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + one_file +---------- + t +(1 row) + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); + equal_lsn_ok +-------------- + t +(1 row) + +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); + default_end_ok +---------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +ERROR: WAL start LSN must be less than end LSN +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); + future_end_ok +--------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files('0/0', '0/1'); +ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -183,6 +231,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -213,6 +268,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users. GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) @@ -223,6 +285,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -251,6 +315,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) @@ -259,6 +330,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 82ab186c340..6f8b4a6359a 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -20,6 +20,7 @@ install_data( 'pg_walinspect.control', 'pg_walinspect--1.0.sql', 'pg_walinspect--1.0--1.1.sql', + 'pg_walinspect--1.1--1.2.sql', kwargs: contrib_data_args, ) @@ -37,4 +38,9 @@ tests += { 'regress_args': ['--temp-config', files('walinspect.conf')], 'runningcheck': false, }, + 'tap': { + 'tests': [ + 't/001_timeline.pl', + ], + }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql new file mode 100644 index 00000000000..9cc9523b4dc --- /dev/null +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -0,0 +1,22 @@ +/* contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql */ + +-- complain if script is sourced in psql, rather than via ALTER EXTENSION +\echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit + +-- +-- pg_get_wal_files() +-- +CREATE FUNCTION pg_get_wal_files( + IN start_lsn pg_lsn, + IN end_lsn pg_lsn DEFAULT NULL, + OUT wal_file text, + OUT segment_start_lsn pg_lsn, + OUT segment_end_lsn pg_lsn +) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'pg_get_wal_files' +LANGUAGE C PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO pg_read_server_files; diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index a172f9e2b40..85d76fe37c4 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -13,6 +13,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/timeline.h" #include "access/xlog.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" @@ -22,6 +23,7 @@ #include "funcapi.h" #include "miscadmin.h" #include "port/pg_bitutils.h" +#include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" @@ -38,6 +40,7 @@ PG_MODULE_MAGIC_EXT( ); PG_FUNCTION_INFO_V1(pg_get_wal_block_info); +PG_FUNCTION_INFO_V1(pg_get_wal_files); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -45,7 +48,7 @@ PG_FUNCTION_INFO_V1(pg_get_wal_stats); PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal); static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn); -static XLogRecPtr GetCurrentLSN(void); +static XLogRecPtr GetCurrentLSN(TimeLineID *current_tli); static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn); static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader); static void GetWALRecordInfo(XLogReaderState *record, Datum *values, @@ -68,11 +71,21 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Metadata for one retained segment in the current timeline's history. */ +typedef struct WalTimeSegment +{ + XLogSegNo segno; /* segment number, used for WAL ordering */ + TimeLineID tli; /* timeline containing this segment */ +} WalTimeSegment; + +static int wal_time_segment_cmp(const void *a, const void *b); +static WalTimeSegment *GetWalTimeSegments(int *nsegments); + /* - * Return the LSN up to which the server has WAL. + * Return the LSN up to which the server has WAL, and optionally its timeline. */ static XLogRecPtr -GetCurrentLSN(void) +GetCurrentLSN(TimeLineID *current_tli) { XLogRecPtr curr_lsn; @@ -81,9 +94,15 @@ GetCurrentLSN(void) * callback read_local_xlog_page_no_wait does. */ if (!RecoveryInProgress()) - curr_lsn = GetFlushRecPtr(NULL); + curr_lsn = GetFlushRecPtr(current_tli); else - curr_lsn = GetXLogReplayRecPtr(NULL); + { + curr_lsn = GetXLogReplayRecPtr(current_tli); + if (current_tli != NULL && *current_tli == 0) + { + *current_tli = GetWALInsertionTimeLineIfSet(); + } + } Assert(XLogRecPtrIsValid(curr_lsn)); @@ -190,6 +209,94 @@ ReadNextXLogRecord(XLogReaderState *xlogreader) return record; } +/* qsort comparator that puts segments in WAL order. */ +static int +wal_time_segment_cmp(const void *a, const void *b) +{ + const WalTimeSegment *seg1 = (const WalTimeSegment *) a; + const WalTimeSegment *seg2 = (const WalTimeSegment *) b; + + if (seg1->segno < seg2->segno) + return -1; + if (seg1->segno > seg2->segno) + return 1; + return 0; +} + +/* + * Get the WAL segments that form the history of the server's current + * timeline. Other timelines may have files with the same segment number in + * pg_wal, so choose the timeline that is valid at the end of each segment, + * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. + */ +static WalTimeSegment * +GetWalTimeSegments(int *nsegments) +{ + WalTimeSegment *segments = NULL; + DIR *dir; + struct dirent *de; + List *history; + XLogRecPtr current_lsn; + TimeLineID current_tli; + int allocated = 0; + int count = 0; + + current_lsn = GetCurrentLSN(&current_tli); + + history = readTimeLineHistory(current_tli); + dir = AllocateDir(XLOGDIR); + while ((de = ReadDir(dir, XLOGDIR)) != NULL) + { + TimeLineID file_tli; + XLogSegNo segno; + XLogRecPtr seg_start; + XLogRecPtr seg_end; + + if (!IsXLogFileName(de->d_name)) + continue; + + XLogFromFileName(de->d_name, &file_tli, &segno, wal_segment_size); + seg_start = segno * wal_segment_size; + seg_end = seg_start + wal_segment_size - 1; + + if (seg_start >= current_lsn || + file_tli != tliOfPointInHistory(seg_end, history)) + continue; + + if (count == allocated) + { + allocated = allocated ? allocated * 2 : 16; + if (segments == NULL) + segments = palloc_array(WalTimeSegment, allocated); + else + segments = repalloc_array(segments, WalTimeSegment, allocated); + } + + MemSet(&segments[count], 0, sizeof(WalTimeSegment)); + segments[count].segno = segno; + segments[count].tli = file_tli; + count++; + } + FreeDir(dir); + list_free_deep(history); + + if (count == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no retained WAL segments are available")); + + qsort(segments, count, sizeof(WalTimeSegment), wal_time_segment_cmp); + +#ifdef USE_ASSERT_CHECKING + /* A segment number identifies at most one file in this timeline history. */ + for (int i = 1; i < count; i++) + Assert(segments[i - 1].segno != segments[i].segno); +#endif + + *nsegments = count; + return segments; +} + /* * Output values that make up a row describing caller's WAL record. * @@ -483,7 +590,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) HeapTuple tuple; lsn = PG_GETARG_LSN(0); - curr_lsn = GetCurrentLSN(); + curr_lsn = GetCurrentLSN(NULL); if (lsn > curr_lsn) ereport(ERROR, @@ -516,6 +623,100 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * List the retained WAL segment files intersecting [start_lsn, end_lsn), or + * the segment containing start_lsn when end_lsn is omitted or equal to it. + * Report complete segment boundaries, not the range clipped to the inputs. + */ +Datum +pg_get_wal_files(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_FILES_COLS 3 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + XLogRecPtr current_lsn; + XLogSegNo first_segno; + XLogSegNo last_segno; + bool point_lookup; + WalTimeSegment *segments; + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + int nsegments; + int segment_index = 0; + + InitMaterializedSRF(fcinfo, 0); + + /* A NULL start produces an empty set. */ + if (PG_ARGISNULL(0)) + PG_RETURN_VOID(); + + start_lsn = PG_GETARG_LSN(0); + end_lsn = PG_ARGISNULL(1) ? start_lsn : PG_GETARG_LSN(1); + + if (start_lsn < XLOG_BLCKSZ) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not read WAL at LSN %X/%08X", + LSN_FORMAT_ARGS(start_lsn))); + + ValidateInputLSNs(start_lsn, &end_lsn); + point_lookup = start_lsn == end_lsn; + current_lsn = GetCurrentLSN(NULL); + + /* + * At a segment boundary, the current LSN can point just past the last + * retained WAL byte, before the next segment file has been created. In + * that case, return the preceding segment containing the available WAL. + */ + if (point_lookup && start_lsn == current_lsn && + XLogSegmentOffset(start_lsn, wal_segment_size) == 0) + XLByteToPrevSeg(start_lsn, first_segno, wal_segment_size); + else + XLByteToSeg(start_lsn, first_segno, wal_segment_size); + if (point_lookup) + last_segno = first_segno; + else + XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); + segments = GetWalTimeSegments(&nsegments); + + for (XLogSegNo segno = first_segno;; segno++) + { + Datum values[PG_GET_WAL_FILES_COLS]; + bool nulls[PG_GET_WAL_FILES_COLS] = {0}; + XLogRecPtr segment_start_lsn = segno * wal_segment_size; + XLogRecPtr segment_end_lsn = segment_start_lsn + wal_segment_size; + char fname[MAXFNAMELEN]; + + while (segment_index < nsegments && + segments[segment_index].segno < segno) + segment_index++; + + if (segment_index >= nsegments || + segments[segment_index].segno != segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the requested range is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) segno)); + + XLogFileName(fname, segments[segment_index].tli, segno, + wal_segment_size); + values[0] = CStringGetTextDatum(fname); + values[1] = LSNGetDatum(segment_start_lsn); + values[2] = LSNGetDatum(segment_end_lsn); + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, + values, nulls); + + if (segno == last_segno) + break; + + CHECK_FOR_INTERRUPTS(); + } + + pfree(segments); + PG_RETURN_VOID(); +#undef PG_GET_WAL_FILES_COLS +} + /* * Validate start and end LSNs coming from the function inputs. * @@ -525,7 +726,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn) { - XLogRecPtr curr_lsn = GetCurrentLSN(); + XLogRecPtr curr_lsn = GetCurrentLSN(NULL); if (start_lsn > curr_lsn) ereport(ERROR, @@ -831,7 +1032,7 @@ Datum pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); if (start_lsn > end_lsn) ereport(ERROR, @@ -849,7 +1050,7 @@ Datum pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); bool stats_per_record = PG_GETARG_BOOL(1); if (start_lsn > end_lsn) diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control index efa3cb2cfe5..5f574b865bb 100644 --- a/contrib/pg_walinspect/pg_walinspect.control +++ b/contrib/pg_walinspect/pg_walinspect.control @@ -1,5 +1,5 @@ # pg_walinspect extension comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log' -default_version = '1.1' +default_version = '1.2' module_pathname = '$libdir/pg_walinspect' relocatable = true diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql index e35c4f3ed15..b935a067b3e 100644 --- a/contrib/pg_walinspect/sql/oldextversions.sql +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -27,10 +27,11 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; --- List what version 1.1 contains. +-- List what the current version contains. \dx+ pg_walinspect SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 1e64a22d29a..3176a17d855 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -51,6 +51,30 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +SELECT * FROM pg_get_wal_files('0/0', '0/1'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -110,6 +134,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; @@ -122,6 +148,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE pg_read_server_files FROM regress_pg_walinspect; @@ -134,6 +162,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes @@ -143,6 +173,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; @@ -152,6 +184,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up diff --git a/contrib/pg_walinspect/t/001_timeline.pl b/contrib/pg_walinspect/t/001_timeline.pl new file mode 100644 index 00000000000..959fe485268 --- /dev/null +++ b/contrib/pg_walinspect/t/001_timeline.pl @@ -0,0 +1,74 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test that pg_get_wal_files() follows the current timeline's history across +# a promotion. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +sub test_timeline_history +{ + my ($suffix, $archive_mode) = @_; + my $primary = PostgreSQL::Test::Cluster->new("primary_$suffix"); + my $standby = PostgreSQL::Test::Cluster->new("standby_$suffix"); + + $primary->init(allows_streaming => 1); + $primary->append_conf('postgresql.conf', "autovacuum = off\n"); + if ($archive_mode) + { + $primary->append_conf( + 'postgresql.conf', + "archive_mode = on\narchive_command = 'true'\n"); + } + $primary->start; + $primary->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + my $boundary_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_lsn()'); + my $boundary_offset = $primary->safe_psql('postgres', + "SELECT file_offset FROM pg_walfile_name_offset('$boundary_lsn')"); + is($boundary_offset, '0', + "WAL is at a segment boundary with archive_mode=$archive_mode"); + my $boundary_files = $primary->safe_psql('postgres', + "SELECT count(*) FROM pg_get_wal_files('$boundary_lsn')"); + is($boundary_files, '1', + "point lookup works at a segment boundary with archive_mode=$archive_mode"); + + my $backup_name = "backup_$suffix"; + $primary->backup($backup_name); + $standby->init_from_backup($primary, $backup_name, has_streaming => 1); + $standby->append_conf('postgresql.conf', "wal_keep_size = '64MB'\n"); + $standby->start; + + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (1)'); + my $start_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (2)'); + $primary->wait_for_catchup($standby); + + $standby->promote; + $standby->safe_psql('postgres', 'INSERT INTO test_table VALUES (3)'); + + my $result = $standby->safe_psql( + 'postgres', qq{ +SELECT count(DISTINCT left(wal_file, 8)) = 2, + min(left(wal_file, 8)) = '00000001', + max(left(wal_file, 8)) = '00000002' +FROM pg_get_wal_files('$start_lsn', pg_current_wal_lsn())}); + is($result, 't|t|t', + "WAL files span the timeline switch with archive_mode=$archive_mode"); + + $standby->stop; + $primary->stop; + return; +} + +test_timeline_history('archive_off', 0); +test_timeline_history('archive_on', 1); + +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 79c3ead40bc..2f708b32c60 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -34,9 +34,8 @@ </note> <tip> <para> - All of the <filename>pg_walinspect</filename> functions that show - information about records that fall within a certain LSN range are - permissive about accepting <replaceable>end_lsn</replaceable> + All of the <filename>pg_walinspect</filename> functions that accept an LSN + range are permissive about accepting <replaceable>end_lsn</replaceable> arguments that are after the server's current LSN. Using an <replaceable>end_lsn</replaceable> <quote>from the future</quote> will not raise an error. @@ -60,6 +59,42 @@ <title>General Functions</title> <variablelist> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-files"> + <term> + <function>pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record</function> + </term> + + <listitem> + <para> + If <replaceable>end_lsn</replaceable> is specified and differs from + <replaceable>start_lsn</replaceable>, returns each retained WAL segment + file that intersects the half-open WAL range + [<replaceable>start_lsn</replaceable>, + <replaceable>end_lsn</replaceable>). If + <replaceable>end_lsn</replaceable> is omitted, is null, or equals + <replaceable>start_lsn</replaceable>, returns the segment containing + <replaceable>start_lsn</replaceable>. The + <structfield>wal_file</structfield> column contains the file name, while + <structfield>segment_start_lsn</structfield> and + <structfield>segment_end_lsn</structfield> contain the complete segment + boundaries rather than boundaries clipped to the requested range. + If a point lookup specifies the current WAL LSN and it lies exactly on + a segment boundary, the preceding segment is returned, since the next + segment might not have been created yet. This differs from + <function>pg_walfile_name</function>, which performs a pure calculation + that maps an LSN to a WAL file name without checking whether the file + exists, and therefore maps such a boundary LSN to the next segment. + </para> + + <para> + File names use the timeline valid for each segment in the current + timeline's history. This allows a returned range to cross a timeline + switch. The function reports an error if any required segment file is + not currently retained in <filename>pg_wal</filename>. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d001f56efae..3ab48bd75ca 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3430,6 +3430,7 @@ WalSummarizerData WalSummaryFile WalSummaryIO WalTimeSample +WalTimeSegment WalUsage WalWriteMethod WalWriteMethodOps -- 2.50.1 (Apple Git-155) =
v4-0002-pg_walinspect-add-function-to-locate-WAL-by-time.patch (application/octet-stream, 60.1K) download From c638733f9456276a73cb14a628992e939325b3f9 Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v4 2/2] pg_walinspect: add function to locate WAL by time Add pg_get_wal_location_at_time(), which uses timestamped WAL records to locate a useful WAL range around a requested wall-clock time. Use an approximate binary search over retained segments and allow callers to widen the requested interval when timestamp reordering makes the initial result insufficient. Move WAL record timestamp extraction from recovery into xlogreader so pg_walinspect can recognize the same timestamp-bearing record types without duplicating that logic. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- .../pg_walinspect/expected/oldextversions.out | 9 +- .../pg_walinspect/expected/pg_walinspect.out | 164 ++++++ contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 21 + contrib/pg_walinspect/pg_walinspect.c | 485 +++++++++++++++++- contrib/pg_walinspect/sql/pg_walinspect.sql | 121 +++++ contrib/pg_walinspect/t/002_time_search.pl | 98 ++++ doc/src/sgml/pgwalinspect.sgml | 117 +++++ src/backend/access/transam/xlogreader.c | 36 ++ src/backend/access/transam/xlogrecovery.c | 48 +- src/include/access/xlogreader.h | 3 + src/tools/pgindent/typedefs.list | 1 + 12 files changed, 1055 insertions(+), 49 deletions(-) create mode 100644 contrib/pg_walinspect/t/002_time_search.pl diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 311017d66ca..6228ad9ef39 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -51,15 +51,16 @@ ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; -- List what the current version contains. \dx+ pg_walinspect - Objects in extension "pg_walinspect" - Object description -------------------------------------------------------- + Objects in extension "pg_walinspect" + Object description +---------------------------------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) function pg_get_wal_files(pg_lsn,pg_lsn) + function pg_get_wal_location_at_time(timestamp with time zone,interval,interval) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(5 rows) +(6 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index 34864c65ec7..341bb543295 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -139,6 +139,145 @@ FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + timestamped_ok +---------------- + t +(1 row) + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | end_ok +----------+-------- + t | t +(1 row) + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + start_ok | current_end_ok | end_lsn_ok | lsn_ok | records_ok +----------+----------------+------------+--------+------------ + t | t | t | t | t +(1 row) + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; + defaults_ok +------------- + t +(1 row) + +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +ERROR: before interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +ERROR: after interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +ERROR: could not locate a WAL time boundary +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); +ERROR: requested time follows the available WAL range +-- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset @@ -231,6 +370,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no has_function_privilege @@ -268,6 +414,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -285,6 +438,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -315,6 +470,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -330,6 +492,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 6f8b4a6359a..79e94e33e4d 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -41,6 +41,7 @@ tests += { 'tap': { 'tests': [ 't/001_timeline.pl', + 't/002_time_search.pl', ], }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql index 9cc9523b4dc..f72d99a3ceb 100644 --- a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -3,6 +3,27 @@ -- complain if script is sourced in psql, rather than via ALTER EXTENSION \echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit +-- +-- pg_get_wal_location_at_time() +-- +CREATE FUNCTION pg_get_wal_location_at_time( + IN target_time timestamptz, + IN before interval DEFAULT '1 minute', + IN after interval DEFAULT '1 minute', + OUT start_timestamp timestamptz, + OUT start_lsn pg_lsn, + OUT end_timestamp timestamptz, + OUT end_lsn pg_lsn +) +AS 'MODULE_PATHNAME', 'pg_get_wal_location_at_time' +LANGUAGE C STRICT PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) + TO pg_read_server_files; + -- -- pg_get_wal_files() -- diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index 85d76fe37c4..09329c17972 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -12,6 +12,8 @@ */ #include "postgres.h" +#include <sys/stat.h> + #include "access/htup_details.h" #include "access/timeline.h" #include "access/xlog.h" @@ -26,7 +28,9 @@ #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgrprotos.h" #include "utils/pg_lsn.h" +#include "utils/timestamp.h" #include "utils/tuplestore.h" /* @@ -41,6 +45,7 @@ PG_MODULE_MAGIC_EXT( PG_FUNCTION_INFO_V1(pg_get_wal_block_info); PG_FUNCTION_INFO_V1(pg_get_wal_files); +PG_FUNCTION_INFO_V1(pg_get_wal_location_at_time); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -71,15 +76,52 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Selected WAL position and timestamp for one time boundary. */ +typedef struct WalTimeBoundary +{ + bool found; /* matching record was found */ + TimestampTz time; /* record's local event time */ + XLogRecPtr lsn; /* record's start LSN */ + int segment_index; /* index of the segment containing the record */ +} WalTimeBoundary; + /* Metadata for one retained segment in the current timeline's history. */ typedef struct WalTimeSegment { XLogSegNo segno; /* segment number, used for WAL ordering */ TimeLineID tli; /* timeline containing this segment */ + bool scanned; /* segment has already been decoded */ + bool has_timestamp; /* segment contains timestamped records */ + TimestampTz min_time; /* minimum record timestamp in segment */ + TimestampTz max_time; /* maximum record timestamp in segment */ + WalTimeBoundary lower; /* best lower anchor in segment */ + WalTimeBoundary upper; /* best upper anchor in segment */ } WalTimeSegment; static int wal_time_segment_cmp(const void *a, const void *b); -static WalTimeSegment *GetWalTimeSegments(int *nsegments); +static WalTimeSegment *GetWalTimeSegments(int *nsegments, + XLogRecPtr *current_lsn_out); +static int FindWalTimeCandidate(WalTimeSegment *segments, + int *search_segments, + int *nsearch_segments, + TimestampTz target_time, + bool search_left, + TimestampTz wanted_start, + TimestampTz wanted_end, + int *candidate_position, + int *nscanned); +static void ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, + TimestampTz wanted_end); +static void MergeWalTimeSegment(WalTimeSegment *segment, + WalTimeBoundary *lower, + WalTimeBoundary *upper); +static void ProbeLowerBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); +static void ProbeUpperBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); /* * Return the LSN up to which the server has WAL, and optionally its timeline. @@ -223,6 +265,46 @@ wal_time_segment_cmp(const void *a, const void *b) return 0; } +/* + * Probe a timestamped record at or before a requested time boundary. Among + * records examined so far, prefer the latest time, breaking ties in favor + * of the earlier WAL record. The caller only passes qualifying records, so + * a greater timestamp is closer to the requested boundary. + */ +static void +ProbeLowerBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time > boundary->time || + (time == boundary->time && lsn < boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + +/* + * Probe a timestamped record at or after a requested time boundary. Among + * records examined so far, prefer the earliest time, breaking ties in favor of + * the later WAL record. The caller only passes qualifying records, so a + * smaller timestamp is closer to the requested boundary. + */ +static void +ProbeUpperBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time < boundary->time || + (time == boundary->time && lsn > boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + /* * Get the WAL segments that form the history of the server's current * timeline. Other timelines may have files with the same segment number in @@ -230,7 +312,7 @@ wal_time_segment_cmp(const void *a, const void *b) * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. */ static WalTimeSegment * -GetWalTimeSegments(int *nsegments) +GetWalTimeSegments(int *nsegments, XLogRecPtr *current_lsn_out) { WalTimeSegment *segments = NULL; DIR *dir; @@ -242,6 +324,8 @@ GetWalTimeSegments(int *nsegments) int count = 0; current_lsn = GetCurrentLSN(&current_tli); + if (current_lsn_out != NULL) + *current_lsn_out = current_lsn; history = readTimeLineHistory(current_tli); dir = AllocateDir(XLOGDIR); @@ -297,6 +381,140 @@ GetWalTimeSegments(int *nsegments) return segments; } +/* + * Approximately locate the segment containing target_time by binary search. + * Record timestamps are expected to generally increase with WAL position. + * Remove segments without timestamped records from the search candidates, + * since they provide no value with which to choose either half. + */ +static int +FindWalTimeCandidate(WalTimeSegment *segments, int *search_segments, + int *nsearch_segments, + TimestampTz target_time, bool search_left, + TimestampTz wanted_start, TimestampTz wanted_end, + int *candidate_position, int *nscanned) +{ + int left = 0; + int right = *nsearch_segments - 1; + int result; + + while (left <= right) + { + int middle = left + (right - left) / 2; + int segment_index = search_segments[middle]; + WalTimeSegment *segment = &segments[segment_index]; + + if (!segment->scanned) + { + ScanWalTimeSegment(segment, segment_index, wanted_start, wanted_end); + (*nscanned)++; + } + + if (!segment->has_timestamp) + { + memmove(&search_segments[middle], &search_segments[middle + 1], + (*nsearch_segments - middle - 1) * sizeof(int)); + (*nsearch_segments)--; + right--; + continue; + } + else if (target_time < segment->min_time) + right = middle - 1; + else if (target_time > segment->max_time) + left = middle + 1; + else + { + if (candidate_position != NULL) + *candidate_position = middle; + return search_segments[middle]; + } + } + + if (*nsearch_segments == 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("No retained WAL segment contains a timestamped record.")); + + if (search_left) + result = Max(right, 0); + else + result = Min(left, *nsearch_segments - 1); + + if (candidate_position != NULL) + *candidate_position = result; + return search_segments[result]; +} + +/* + * Inspect timestamped records beginning in one segment and save its time + * range and best boundary candidates. XLogReadRecord() may read the following + * segment to assemble a record crossing the boundary, but that record is + * attributed to the segment containing its starting LSN. + */ +static void +ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, TimestampTz wanted_end) +{ + XLogRecPtr seg_start = segment->segno * wal_segment_size; + XLogRecPtr seg_end = seg_start + wal_segment_size; + XLogReaderState *xlogreader; + + Assert(!segment->scanned); + segment->scanned = true; + xlogreader = InitXLogReaderState(seg_start); + + while (ReadNextXLogRecord(xlogreader)) + { + TimestampTz record_time; + + if (xlogreader->ReadRecPtr >= seg_end) + break; + + if (!GetXLogRecordTimestamp(xlogreader, &record_time)) + continue; + + if (!segment->has_timestamp) + { + segment->has_timestamp = true; + segment->min_time = segment->max_time = record_time; + } + else + { + segment->min_time = Min(segment->min_time, record_time); + segment->max_time = Max(segment->max_time, record_time); + } + + if (record_time <= wanted_start) + ProbeLowerBoundary(&segment->lower, record_time, + xlogreader->ReadRecPtr, segment_index); + + else if (record_time >= wanted_end) + ProbeUpperBoundary(&segment->upper, record_time, + xlogreader->ReadRecPtr, segment_index); + + CHECK_FOR_INTERRUPTS(); + } + + pfree(xlogreader->private_data); + XLogReaderFree(xlogreader); +} + +/* Merge one segment's boundary candidates into the result. */ +static void +MergeWalTimeSegment(WalTimeSegment *segment, WalTimeBoundary *lower, + WalTimeBoundary *upper) +{ + Assert(segment->scanned); + + if (lower != NULL && segment->lower.found) + ProbeLowerBoundary(lower, segment->lower.time, segment->lower.lsn, + segment->lower.segment_index); + if (upper != NULL && segment->upper.found) + ProbeUpperBoundary(upper, segment->upper.time, segment->upper.lsn, + segment->upper.segment_index); +} + /* * Output values that make up a row describing caller's WAL record. * @@ -623,6 +841,267 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * Locate a WAL region around a wall-clock time range using timestamped WAL + * records. + * + * Locate each boundary by an approximate binary search over retained WAL + * segments, assuming that record timestamps generally increase with WAL + * position. Also examine one preceding segment for the lower boundary and + * one following segment for the upper boundary to allow for limited timestamp + * reordering. The lower boundary uses the latest timestamped record at or + * before the requested start time, while the upper boundary uses the earliest + * one at or after the requested end time. A future upper bound is first + * capped at the current time. If no upper timestamped record is available, + * use the current WAL position. On success, the boundaries form a forward + * WAL range, but are not necessarily the globally closest timestamped records + * to the requested bounds. + */ +Datum +pg_get_wal_location_at_time(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_LOCATION_AT_TIME_COLS 4 + TimestampTz target_time = PG_GETARG_TIMESTAMPTZ(0); + Interval *before = PG_GETARG_INTERVAL_P(1); + Interval *after = PG_GETARG_INTERVAL_P(2); + Interval zero = {0}; + Interval one_day = {.day = 1}; + TimestampTz wanted_start; + TimestampTz wanted_end; + TimestampTz current_time; + XLogRecPtr current_lsn; + XLogSegNo current_segno; + WalTimeSegment *segments; + WalTimeBoundary lower = {0}; + WalTimeBoundary upper = {0}; + Datum values[PG_GET_WAL_LOCATION_AT_TIME_COLS]; + bool nulls[PG_GET_WAL_LOCATION_AT_TIME_COLS] = {0}; + TupleDesc tupdesc; + HeapTuple tuple; + int nsegments; + int *search_segments; + int nsearch_segments; + int lower_candidate; + int upper_candidate; + int lower_candidate_position; + int nupper_search_segments; + int lower_start; + int upper_end; + int upper_segment_index; + int nscanned = 0; + bool upper_bound_capped = false; + char lower_fname[MAXFNAMELEN]; + char upper_fname[MAXFNAMELEN]; + + if (TIMESTAMP_NOT_FINITE(target_time)) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("target time must be finite")); + if (INTERVAL_NOT_FINITE(before) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must not exceed one day")); + if (INTERVAL_NOT_FINITE(after) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must not exceed one day")); + + /* The timestamp functions provide the normal overflow checks. */ + wanted_start = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_mi_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(before))); + wanted_end = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_pl_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(after))); + current_time = GetCurrentTimestamp(); + + /* Cap a future upper bound at the time currently available. */ + if (wanted_end > current_time) + { + if (wanted_start >= current_time) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("The requested lower bound is not earlier than the current time.")); + + wanted_end = current_time; + upper_bound_capped = true; + } + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + segments = GetWalTimeSegments(&nsegments, &current_lsn); + XLByteToPrevSeg(current_lsn, current_segno, wal_segment_size); + + /* + * A capped upper bound may need the current WAL position as its endpoint. + * Detect a missing WAL tail before decoding any candidate segments. + */ + if (upper_bound_capped && + segments[nsegments - 1].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The current WAL LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(current_lsn), + (uint64) current_segno)); + + search_segments = palloc_array(int, nsegments); + for (int i = 0; i < nsegments; i++) + search_segments[i] = i; + nsearch_segments = nsegments; + lower_candidate = FindWalTimeCandidate(segments, search_segments, + &nsearch_segments, wanted_start, + true, wanted_start, wanted_end, + &lower_candidate_position, + &nscanned); + + /* Search for the later upper boundary from the lower candidate onward. */ + nupper_search_segments = nsearch_segments - lower_candidate_position; + upper_candidate = FindWalTimeCandidate(segments, + search_segments + lower_candidate_position, + &nupper_search_segments, wanted_end, + false, wanted_start, wanted_end, + NULL, + &nscanned); + XLogFileName(lower_fname, segments[lower_candidate].tli, + segments[lower_candidate].segno, wal_segment_size); + XLogFileName(upper_fname, segments[upper_candidate].tli, + segments[upper_candidate].segno, wal_segment_size); + ereport(DEBUG1, + errmsg_internal("WAL time search selected lower segment %s and upper segment %s from %d retained segments", + lower_fname, upper_fname, + nsegments)); + + /* + * Allow for limited timestamp reordering by examining one preceding + * segment for the lower boundary and one following segment for the upper + * boundary. Do not extend across a missing segment. + */ + lower_start = lower_candidate; + if (lower_start > 0 && + segments[lower_start - 1].segno + 1 == segments[lower_start].segno) + lower_start--; + upper_end = upper_candidate; + if (upper_end + 1 < nsegments && + segments[upper_end].segno + 1 == segments[upper_end + 1].segno) + upper_end++; + + for (int i = lower_start; i <= lower_candidate; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], &lower, NULL); + } + for (int i = upper_candidate; i <= upper_end; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], NULL, &upper); + } + + ereport(DEBUG1, + errmsg_internal("WAL time search decoded %d of %d retained segments", + nscanned, nsegments)); + + if (!lower.found) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("The WAL search did not find a timestamped record at or before the requested lower bound.")); + + /* + * There may be no timestamped record after the requested upper bound on a + * quiet system. In that case, use the current WAL position so that the + * returned range still includes all WAL currently available. + */ + if (!upper.found) + { + upper.lsn = current_lsn; + + /* + * The last enumerated segment should contain current_lsn. Verify + * this because pg_wal can change while it is being enumerated; + * otherwise the continuity check below would not cover a missing WAL + * tail. + */ + upper_segment_index = nsegments - 1; + if (segments[upper_segment_index].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) current_segno)); + } + else + upper_segment_index = upper.segment_index; + + if (lower.lsn >= upper.lsn) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("could not find a valid WAL range for the requested time window"), + errdetail("The matching lower boundary does not precede the upper boundary in WAL.")); + + Assert(lower.segment_index >= 0 && lower.segment_index < nsegments); + Assert(upper_segment_index >= 0 && upper_segment_index < nsegments); + for (int i = lower.segment_index + 1; i <= upper_segment_index; i++) + { + /* + * The returned LSN range must be continuously readable from pg_wal; + * otherwise, functions such as pg_get_wal_records_info() would fail + * when scanning the range across a missing segment. + */ + if (segments[i - 1].segno + 1 != segments[i].segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) (segments[i - 1].segno + 1))); + } + + values[0] = TimestampTzGetDatum(lower.time); + values[1] = LSNGetDatum(lower.lsn); + if (upper.found) + values[2] = TimestampTzGetDatum(upper.time); + else + nulls[2] = true; + values[3] = LSNGetDatum(upper.lsn); + + pfree(segments); + tuple = heap_form_tuple(tupdesc, values, nulls); + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +#undef PG_GET_WAL_LOCATION_AT_TIME_COLS +} + /* * List the retained WAL segment files intersecting [start_lsn, end_lsn), or * the segment containing start_lsn when end_lsn is omitted or equal to it. @@ -676,7 +1155,7 @@ pg_get_wal_files(PG_FUNCTION_ARGS) last_segno = first_segno; else XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); - segments = GetWalTimeSegments(&nsegments); + segments = GetWalTimeSegments(&nsegments, NULL); for (XLogSegNo segno = first_segno;; segno++) { diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 3176a17d855..4b37a25fa41 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -75,6 +75,117 @@ SELECT count(*) = 1 AS future_end_ok FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== + +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); + +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); + +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); + +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -134,6 +245,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no @@ -148,6 +261,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -162,6 +277,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; @@ -173,6 +290,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -184,6 +303,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; diff --git a/contrib/pg_walinspect/t/002_time_search.pl b/contrib/pg_walinspect/t/002_time_search.pl new file mode 100644 index 00000000000..96a5c06eb76 --- /dev/null +++ b/contrib/pg_walinspect/t/002_time_search.pl @@ -0,0 +1,98 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test the bounded binary search for WAL time boundaries across segments. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $node = PostgreSQL::Test::Cluster->new('time_search'); +$node->init(extra => ['--wal-segsize=1']); +$node->append_conf('postgresql.conf', "autovacuum = off\n"); +$node->start; +$node->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); +$node->safe_psql('postgres', 'SELECT pg_switch_wal()'); + +my ($lower_time, $upper_time); +for my $i (0 .. 8) +{ + $node->safe_psql('postgres', "INSERT INTO test_table VALUES ($i)"); + $lower_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 3; + $upper_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 5; + $node->safe_psql('postgres', 'SELECT pg_switch_wal()') if $i < 8; +} + +my $log_offset = -s $node->logfile; +my $result = $node->safe_psql( + 'postgres', qq{ +SET log_min_messages = debug1; +SELECT start_timestamp <= '$lower_time'::timestamptz - interval '1 microsecond', + end_timestamp >= '$upper_time'::timestamptz, + start_lsn < end_lsn, + pg_walfile_name(start_lsn) <> pg_walfile_name(end_lsn) +FROM pg_get_wal_location_at_time('$lower_time', + interval '1 microsecond', + '$upper_time'::timestamptz - + '$lower_time'::timestamptz)}); +is($result, 't|t|t|t', 'binary search finds outer boundaries across WAL segments'); + +my $log = slurp_file($node->logfile, $log_offset); +$log =~ /WAL time search decoded (\d+) of (\d+) retained segments/ + or die "could not find WAL time search statistics in server log"; +cmp_ok($1, '<', $2, 'binary search does not decode all retained WAL'); + +# A window containing no timestamped record is enclosed by the timestamped +# records immediately outside it. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS empty_window_start \gset +SELECT count(*) AS ignored FROM generate_series(1, 100000) \gset +SELECT clock_timestamp() AS empty_window_end \gset +INSERT INTO test_table VALUES (9); + +SELECT location.start_timestamp < :'empty_window_start'::timestamptz, + location.end_timestamp > :'empty_window_end'::timestamptz, + location.start_lsn < location.end_lsn +FROM pg_get_wal_location_at_time( + :'empty_window_start', + interval '1 microsecond', + :'empty_window_end'::timestamptz - + :'empty_window_start'::timestamptz) AS location;}); +is($result, 't|t|t', + 'outer anchors enclose a window with no timestamped record'); + +# With no later timestamped record, use the current WAL position as the upper +# boundary. The returned range must include WAL generated before the only +# commit record in the requested window. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS window_start \gset +BEGIN; +SELECT pg_current_wal_insert_lsn() AS delete_lsn \gset +DELETE FROM test_table WHERE a = 5; +COMMIT; +SELECT clock_timestamp() AS window_end \gset + +SELECT location.start_timestamp <= + :'window_start'::timestamptz - interval '1 microsecond', + location.end_timestamp IS NULL, + location.start_lsn < :'delete_lsn'::pg_lsn, + location.end_lsn > :'delete_lsn'::pg_lsn, + EXISTS (SELECT + FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'DELETE') +FROM pg_get_wal_location_at_time( + :'window_start', + interval '1 microsecond', + :'window_end'::timestamptz - :'window_start'::timestamptz) AS location;}); +is($result, 't|t|t|t|t', + 'current WAL fallback includes changes preceding the only commit in the window'); + +$node->stop; +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 2f708b32c60..9d87cb6d4fe 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -95,6 +95,123 @@ </listitem> </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-location-at-time"> + <term> + <function>pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute') returns record</function> + </term> + + <listitem> + <para> + Locates a nearby WAL region using records containing local event + timestamps as historical time anchors. The requested time range extends + from <replaceable>target_time</replaceable> minus + <replaceable>before</replaceable> through + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable>. It returns + <structfield>start_timestamp</structfield>, + <structfield>start_lsn</structfield>, + <structfield>end_timestamp</structfield>, and + <structfield>end_lsn</structfield>. The effective upper bound is the + earlier of <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> and the current time. On successful + return, the following relationships hold: +<programlisting> +start_timestamp &lt;= target_time - before +end_timestamp &gt;= effective upper bound, or is null +start_lsn &lt; end_lsn +</programlisting> + The start boundary is the latest timestamped record found at or before + the requested start time, and the end boundary is the earliest + timestamped record found at or after the requested end time. If + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> is later than the current time, the + requested upper bound is first capped at the current time. If no + timestamped record is available at or after the upper bound, + <structfield>end_lsn</structfield> is the current flush or replay + position and <structfield>end_timestamp</structfield> is null. Both + interval arguments must be greater than zero and not greater than one + day. + </para> + + <para> + Most WAL records do not contain a wall-clock timestamp. Historical + anchors can be transaction <literal>COMMIT</literal>, + <literal>COMMIT PREPARED</literal>, <literal>ABORT</literal>, + or <literal>ABORT PREPARED</literal> records, as well as restore point + records. These are the WAL record types from which PostgreSQL recovery + can extract timestamps. Their timestamps describe the corresponding + event; they do not assign timestamps to heap, index, or other records. + </para> + + <para> + In particular, a commit timestamp describes when a transaction + committed, not when its earlier changes generated WAL. For example, a + transaction might update a row at 14:00 and commit at 15:00; the heap + WAL was generated around 14:00, while the commit record is timestamped + 15:00. Using timestamped records outside the requested time window as + boundaries helps include the WAL generated by transactions that commit + within the window. However, the returned LSNs do not identify the exact + generation time of heap, index, or other WAL records, and a transaction + can have generated WAL before the selected start boundary. On a cluster + with little timestamped activity, the returned timestamps can be far + from the requested bounds. + </para> + + <para> + The function assumes that WAL record timestamps generally increase with + WAL position and uses an approximate binary search to locate each time + boundary. WAL record timestamps are not guaranteed to follow WAL order, + for example because of concurrent activity, clock adjustments, or + delayed WAL insertion. Such timestamp reordering can cause relevant + records to fall outside the returned range. Therefore, the returned + records are not guaranteed to be the globally closest timestamped + records to either requested time bound. If the returned range does not + contain the records of interest, the user can widen the search range and + repeat the search. If the matching lower boundary follows the upper + boundary in WAL, the function reports an error rather than returning a + range with the LSNs reversed. + </para> + + <para> + Only WAL that is currently retained in <filename>pg_wal</filename> can + be searched. Unlike the other <application>pg_walinspect</application> + functions, this function may inspect retained WAL from the current + timeline's history as well as the current timeline. The function + reports an error when a lower boundary cannot be found by the bounded + search or when the resulting range would cross a missing WAL segment. + When an upper timestamped boundary cannot be found, the current WAL + position is returned instead, as described above. + </para> + + <para> + For example, a DBA can locate WAL around activity near a given time and + then inspect the returned range: +<programlisting> +SELECT * +FROM pg_get_wal_location_at_time( + '2026-08-01 14:35:00+08', + before =&gt; '10 minutes', + after =&gt; '10 minutes'); + +SELECT * +FROM pg_get_wal_files('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); + +SELECT * +FROM pg_get_wal_records_info('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); +</programlisting> + The corresponding retained WAL files can be obtained with + <function>pg_get_wal_files</function>. To widen the search range, + increase <replaceable>before</replaceable> if the relevant activity + might have occurred earlier, or increase + <replaceable>after</replaceable> if it might have occurred later, and + repeat the search. Failure to find a record in the first returned range + does not necessarily mean that the record is absent from retained WAL. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index cc61f830982..08ca9e1b491 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -26,6 +26,7 @@ #endif #include "access/transam.h" +#include "access/xact.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" #include "access/xlogrecord.h" @@ -72,6 +73,41 @@ static void XLogReaderFreeZstdContext(void *arg); */ #define DEFAULT_DECODE_BUFFER_SIZE (64 * 1024) +/* + * Extract timestamp from WAL record. + * + * If the record contains a timestamp, returns true, and saves the timestamp + * in *recordXtime. If the record type has no timestamp, returns false. + * Currently, only transaction commit/abort records and restore points contain + * timestamps. + */ +bool +GetXLogRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) +{ + uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; + uint8 xact_info = info & XLOG_XACT_OPMASK; + uint8 rmid = XLogRecGetRmid(record); + + if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) + { + *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || + xact_info == XLOG_XACT_COMMIT_PREPARED)) + { + *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || + xact_info == XLOG_XACT_ABORT_PREPARED)) + { + *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; + return true; + } + return false; +} + /* * Construct a string in state->errormsg_buf explaining what's wrong with * the current record being read. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 54aaec9529f..06a5b10d092 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -356,7 +356,6 @@ static void xlog_outrec(StringInfo buf, XLogReaderState *record); static void xlog_block_info(StringInfo buf, XLogReaderState *record); static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, TimeLineID replayTLI); -static bool getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime); static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); @@ -2405,41 +2404,6 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, } -/* - * Extract timestamp from WAL record. - * - * If the record contains a timestamp, returns true, and saves the timestamp - * in *recordXtime. If the record type has no timestamp, returns false. - * Currently, only transaction commit/abort records and restore points contain - * timestamps. - */ -static bool -getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) -{ - uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; - uint8 xact_info = info & XLOG_XACT_OPMASK; - uint8 rmid = XLogRecGetRmid(record); - - if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) - { - *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || - xact_info == XLOG_XACT_COMMIT_PREPARED)) - { - *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || - xact_info == XLOG_XACT_ABORT_PREPARED)) - { - *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; - return true; - } - return false; -} - /* * Checks whether the current buffer page and backup page stored in the * WAL record are consistent or not. Before comparing the two pages, a @@ -2663,10 +2627,10 @@ recoveryStopsBefore(XLogReaderState *record) /* * Note: we must fetch recordXtime regardless of recoveryTarget setting. - * We don't expect getRecordTimestamp ever to fail, since we already know - * this is a commit or abort record; but test its result anyway. + * We don't expect GetXLogRecordTimestamp ever to fail, since we already + * know this is a commit or abort record; but test its result anyway. */ - if (getRecordTimestamp(record, &recordXtime) && + if (GetXLogRecordTimestamp(record, &recordXtime) && recoveryTarget == RECOVERY_TARGET_TIME) { /* @@ -2747,7 +2711,7 @@ recoveryStopsAfter(XLogReaderState *record) recoveryStopAfter = true; recoveryStopXid = InvalidTransactionId; recoveryStopLSN = InvalidXLogRecPtr; - (void) getRecordTimestamp(record, &recoveryStopTime); + (void) GetXLogRecordTimestamp(record, &recoveryStopTime); strlcpy(recoveryStopName, recordRestorePointData->rp_name, MAXFNAMELEN); ereport(LOG, @@ -2787,7 +2751,7 @@ recoveryStopsAfter(XLogReaderState *record) TransactionId recordXid; /* Update the last applied transaction timestamp */ - if (getRecordTimestamp(record, &recordXtime)) + if (GetXLogRecordTimestamp(record, &recordXtime)) SetLatestXTime(recordXtime); /* Extract the XID of the committed/aborted transaction */ @@ -3006,7 +2970,7 @@ recoveryApplyDelay(XLogReaderState *record) xact_info != XLOG_XACT_COMMIT_PREPARED) return false; - if (!getRecordTimestamp(record, &xtime)) + if (!GetXLogRecordTimestamp(record, &xtime)) return false; delayUntil = TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e48a0db9ce9..43682a28272 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -39,6 +39,7 @@ #endif #include "access/xlogrecord.h" +#include "datatype/timestamp.h" #include "storage/buf.h" /* WALOpenSegment represents a WAL segment being read. */ @@ -462,5 +463,7 @@ extern bool XLogRecGetBlockTagExtended(XLogReaderState *record, uint8 block_id, RelFileLocator *rlocator, ForkNumber *forknum, BlockNumber *blknum, Buffer *prefetch_buffer); +extern bool GetXLogRecordTimestamp(XLogReaderState *record, + TimestampTz *recordXtime); #endif /* XLOGREADER_H */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3ab48bd75ca..fb81d5e862a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3429,6 +3429,7 @@ WalSndState WalSummarizerData WalSummaryFile WalSummaryIO +WalTimeBoundary WalTimeSample WalTimeSegment WalUsage -- 2.50.1 (Apple Git-155) =
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-09 23:18  surya poondla <suryapoondla4@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: surya poondla @ 2026-10-09 23:18 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi Chao,

Thanks for v4. The test fix and the doc sentence look good, and CFBot is
green now, including macOS.

I did a full read-through of v4, especially the boundary logic in 0002.
The binary search, the outer anchors, the extra segment scanned on each
side, and the fallback to the current WAL position all look correct to
me. The tie-breaks and the neighbour scan only ever widen the range, which
is the safe direction.

I have a few small comments:
1. pg_get_wal_records_info(start_lsn, end_lsn) includes the start-anchor
record but not the end-anchor one.
end_lsn is the start LSN of the end-anchor record, and
pg_get_wal_records_info() only returns records that end at or before
end_lsn. So if the transaction of interest committed just after the window,
its COMMIT can be the end anchor, the user sees its changes but
not its commit, and might conclude it never committed. I wouldn't change
end_lsn, since pointing pg_get_wal_record_info() at it to see the anchor
is useful, but a sentence in the docs would help. (This only applies when
an end anchor is found, not when end_lsn falls back to the current WAL
position.)

2. The docs say "Unlike the other pg_walinspect functions, this function
may inspect retained WAL from the current timeline's history as well as
the current timeline." I don't think that's accurate. The existing
functions read through read_local_xlog_page_no_wait(), and
XLogReadDetermineTimeline() already follows the timeline history, choosing
each segment's timeline with tliOfPointInHistory(), the same
rule GetWalTimeSegments() uses. I'd drop the sentence.

3. Nit: "No retained WAL segment contains a timestamped record" is
reported with ERRCODE_INVALID_PARAMETER_VALUE, but nothing is wrong with
the arguments. ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, which the patch
already uses for "no retained WAL segments are available", seems a better
fit.

With these addressed, I think the patch is ready for committer.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-10 03:40  Chao Li <li.evan.chao@gmail.com>
  parent: surya poondla <suryapoondla4@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Chao Li @ 2026-10-10 03:40 UTC (permalink / raw)
  To: surya poondla <suryapoondla4@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>



> On Oct 10, 2026, at 07:18, surya poondla <suryapoondla4@gmail.com> wrote:
> 
> Hi Chao,
> 
> Thanks for v4. The test fix and the doc sentence look good, and CFBot is green now, including macOS.

Thank you very much for the follow up reviewing.

> 
> I did a full read-through of v4, especially the boundary logic in 0002.
> The binary search, the outer anchors, the extra segment scanned on each side, and the fallback to the current WAL position all look correct to
> me. The tie-breaks and the neighbour scan only ever widen the range, which is the safe direction.

Cool!

> 
> I have a few small comments:
> 1. pg_get_wal_records_info(start_lsn, end_lsn) includes the start-anchor record but not the end-anchor one.
> end_lsn is the start LSN of the end-anchor record, and pg_get_wal_records_info() only returns records that end at or before
> end_lsn. So if the transaction of interest committed just after the window, its COMMIT can be the end anchor, the user sees its changes but
> not its commit, and might conclude it never committed. I wouldn't change end_lsn, since pointing pg_get_wal_record_info() at it to see the anchor
> is useful, but a sentence in the docs would help. (This only applies when an end anchor is found, not when end_lsn falls back to the current WAL position.)
> 

This is a good point. Since the requested time range is likely to be only a rough estimate, users may generally be more interested in the surrounding WAL than in the exact endpoint. But I agree that this can sometimes matter.

I considered changing end_lsn to the end-anchor record's ending LSN, but then pg_get_wal_record_info(end_lsn) could no longer be used to inspect the end-anchor record directly. Therefore, I added a tip to the doc explaining this situation and how to include the end-anchor record.

> 2. The docs say "Unlike the other pg_walinspect functions, this function may inspect retained WAL from the current timeline's history as well as
> the current timeline." I don't think that's accurate. The existing functions read through read_local_xlog_page_no_wait(), and
> XLogReadDetermineTimeline() already follows the timeline history, choosing each segment's timeline with tliOfPointInHistory(), the same
> rule GetWalTimeSegments() uses. I'd drop the sentence.

Agreed.

> 
> 3. Nit: "No retained WAL segment contains a timestamped record" is
> reported with ERRCODE_INVALID_PARAMETER_VALUE, but nothing is wrong with
> the arguments. ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE, which the patch
> already uses for "no retained WAL segments are available", seems a better fit.

Agreed.

> 
> With these addressed, I think the patch is ready for committer.
> 
> Regards,
> Surya Poondla

PFA v5: 0001 unchanged; 0002 addressed the new comments.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments: download all (2 files, .tar.gz)

v5-0001-pg_walinspect-add-function-to-list-WAL-files-by-L.patch (application/octet-stream, 29.3K) download From d03866d3efb5a54bfe9f1c95233a638a97dc78e3 Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v5 1/2] pg_walinspect: add function to list WAL files by LSN Add pg_get_wal_files(), which returns the retained WAL segment files intersecting an LSN range, together with each segment's complete LSN boundaries. A point lookup can be requested by omitting end_lsn or making it equal to start_lsn. Choose segment file names from the server's current timeline history so that ranges spanning a timeline switch are handled correctly. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- contrib/pg_walinspect/Makefile | 4 +- .../pg_walinspect/expected/oldextversions.out | 8 +- .../pg_walinspect/expected/pg_walinspect.out | 73 ++++++ contrib/pg_walinspect/meson.build | 6 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 22 ++ contrib/pg_walinspect/pg_walinspect.c | 219 +++++++++++++++++- contrib/pg_walinspect/pg_walinspect.control | 2 +- contrib/pg_walinspect/sql/oldextversions.sql | 5 +- contrib/pg_walinspect/sql/pg_walinspect.sql | 34 +++ contrib/pg_walinspect/t/001_timeline.pl | 74 ++++++ doc/src/sgml/pgwalinspect.sgml | 41 +++- src/tools/pgindent/typedefs.list | 1 + 12 files changed, 470 insertions(+), 19 deletions(-) create mode 100644 contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql create mode 100644 contrib/pg_walinspect/t/001_timeline.pl diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 22090f7716c..b7651231077 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -7,9 +7,11 @@ OBJS = \ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write-Ahead Log" EXTENSION = pg_walinspect -DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql +DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql \ + pg_walinspect--1.1--1.2.sql REGRESS = pg_walinspect oldextversions +TAP_TESTS = 1 REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_walinspect/walinspect.conf diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 89953af3d0e..311017d66ca 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -46,18 +46,20 @@ SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); ERROR: WAL start LSN must be less than current LSN --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; --- List what version 1.1 contains. +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; +-- List what the current version contains. \dx+ pg_walinspect Objects in extension "pg_walinspect" Object description ------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) + function pg_get_wal_files(pg_lsn,pg_lsn) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(4 rows) +(5 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index f955ff5d3c5..34864c65ec7 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -90,6 +90,54 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); t (1 row) +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + file_ok | start_ok | end_ok +---------+----------+-------- + t | t | t +(1 row) + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + one_file +---------- + t +(1 row) + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); + equal_lsn_ok +-------------- + t +(1 row) + +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); + default_end_ok +---------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +ERROR: WAL start LSN must be less than end LSN +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); + future_end_ok +--------------- + t +(1 row) + +SELECT * FROM pg_get_wal_files('0/0', '0/1'); +ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -183,6 +231,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -213,6 +268,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users. GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) @@ -223,6 +285,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -251,6 +315,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) @@ -259,6 +330,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 82ab186c340..6f8b4a6359a 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -20,6 +20,7 @@ install_data( 'pg_walinspect.control', 'pg_walinspect--1.0.sql', 'pg_walinspect--1.0--1.1.sql', + 'pg_walinspect--1.1--1.2.sql', kwargs: contrib_data_args, ) @@ -37,4 +38,9 @@ tests += { 'regress_args': ['--temp-config', files('walinspect.conf')], 'runningcheck': false, }, + 'tap': { + 'tests': [ + 't/001_timeline.pl', + ], + }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql new file mode 100644 index 00000000000..9cc9523b4dc --- /dev/null +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -0,0 +1,22 @@ +/* contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql */ + +-- complain if script is sourced in psql, rather than via ALTER EXTENSION +\echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit + +-- +-- pg_get_wal_files() +-- +CREATE FUNCTION pg_get_wal_files( + IN start_lsn pg_lsn, + IN end_lsn pg_lsn DEFAULT NULL, + OUT wal_file text, + OUT segment_start_lsn pg_lsn, + OUT segment_end_lsn pg_lsn +) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'pg_get_wal_files' +LANGUAGE C PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO pg_read_server_files; diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index a172f9e2b40..85d76fe37c4 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -13,6 +13,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/timeline.h" #include "access/xlog.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" @@ -22,6 +23,7 @@ #include "funcapi.h" #include "miscadmin.h" #include "port/pg_bitutils.h" +#include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" @@ -38,6 +40,7 @@ PG_MODULE_MAGIC_EXT( ); PG_FUNCTION_INFO_V1(pg_get_wal_block_info); +PG_FUNCTION_INFO_V1(pg_get_wal_files); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -45,7 +48,7 @@ PG_FUNCTION_INFO_V1(pg_get_wal_stats); PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal); static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn); -static XLogRecPtr GetCurrentLSN(void); +static XLogRecPtr GetCurrentLSN(TimeLineID *current_tli); static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn); static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader); static void GetWALRecordInfo(XLogReaderState *record, Datum *values, @@ -68,11 +71,21 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Metadata for one retained segment in the current timeline's history. */ +typedef struct WalTimeSegment +{ + XLogSegNo segno; /* segment number, used for WAL ordering */ + TimeLineID tli; /* timeline containing this segment */ +} WalTimeSegment; + +static int wal_time_segment_cmp(const void *a, const void *b); +static WalTimeSegment *GetWalTimeSegments(int *nsegments); + /* - * Return the LSN up to which the server has WAL. + * Return the LSN up to which the server has WAL, and optionally its timeline. */ static XLogRecPtr -GetCurrentLSN(void) +GetCurrentLSN(TimeLineID *current_tli) { XLogRecPtr curr_lsn; @@ -81,9 +94,15 @@ GetCurrentLSN(void) * callback read_local_xlog_page_no_wait does. */ if (!RecoveryInProgress()) - curr_lsn = GetFlushRecPtr(NULL); + curr_lsn = GetFlushRecPtr(current_tli); else - curr_lsn = GetXLogReplayRecPtr(NULL); + { + curr_lsn = GetXLogReplayRecPtr(current_tli); + if (current_tli != NULL && *current_tli == 0) + { + *current_tli = GetWALInsertionTimeLineIfSet(); + } + } Assert(XLogRecPtrIsValid(curr_lsn)); @@ -190,6 +209,94 @@ ReadNextXLogRecord(XLogReaderState *xlogreader) return record; } +/* qsort comparator that puts segments in WAL order. */ +static int +wal_time_segment_cmp(const void *a, const void *b) +{ + const WalTimeSegment *seg1 = (const WalTimeSegment *) a; + const WalTimeSegment *seg2 = (const WalTimeSegment *) b; + + if (seg1->segno < seg2->segno) + return -1; + if (seg1->segno > seg2->segno) + return 1; + return 0; +} + +/* + * Get the WAL segments that form the history of the server's current + * timeline. Other timelines may have files with the same segment number in + * pg_wal, so choose the timeline that is valid at the end of each segment, + * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. + */ +static WalTimeSegment * +GetWalTimeSegments(int *nsegments) +{ + WalTimeSegment *segments = NULL; + DIR *dir; + struct dirent *de; + List *history; + XLogRecPtr current_lsn; + TimeLineID current_tli; + int allocated = 0; + int count = 0; + + current_lsn = GetCurrentLSN(&current_tli); + + history = readTimeLineHistory(current_tli); + dir = AllocateDir(XLOGDIR); + while ((de = ReadDir(dir, XLOGDIR)) != NULL) + { + TimeLineID file_tli; + XLogSegNo segno; + XLogRecPtr seg_start; + XLogRecPtr seg_end; + + if (!IsXLogFileName(de->d_name)) + continue; + + XLogFromFileName(de->d_name, &file_tli, &segno, wal_segment_size); + seg_start = segno * wal_segment_size; + seg_end = seg_start + wal_segment_size - 1; + + if (seg_start >= current_lsn || + file_tli != tliOfPointInHistory(seg_end, history)) + continue; + + if (count == allocated) + { + allocated = allocated ? allocated * 2 : 16; + if (segments == NULL) + segments = palloc_array(WalTimeSegment, allocated); + else + segments = repalloc_array(segments, WalTimeSegment, allocated); + } + + MemSet(&segments[count], 0, sizeof(WalTimeSegment)); + segments[count].segno = segno; + segments[count].tli = file_tli; + count++; + } + FreeDir(dir); + list_free_deep(history); + + if (count == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no retained WAL segments are available")); + + qsort(segments, count, sizeof(WalTimeSegment), wal_time_segment_cmp); + +#ifdef USE_ASSERT_CHECKING + /* A segment number identifies at most one file in this timeline history. */ + for (int i = 1; i < count; i++) + Assert(segments[i - 1].segno != segments[i].segno); +#endif + + *nsegments = count; + return segments; +} + /* * Output values that make up a row describing caller's WAL record. * @@ -483,7 +590,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) HeapTuple tuple; lsn = PG_GETARG_LSN(0); - curr_lsn = GetCurrentLSN(); + curr_lsn = GetCurrentLSN(NULL); if (lsn > curr_lsn) ereport(ERROR, @@ -516,6 +623,100 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * List the retained WAL segment files intersecting [start_lsn, end_lsn), or + * the segment containing start_lsn when end_lsn is omitted or equal to it. + * Report complete segment boundaries, not the range clipped to the inputs. + */ +Datum +pg_get_wal_files(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_FILES_COLS 3 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + XLogRecPtr current_lsn; + XLogSegNo first_segno; + XLogSegNo last_segno; + bool point_lookup; + WalTimeSegment *segments; + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + int nsegments; + int segment_index = 0; + + InitMaterializedSRF(fcinfo, 0); + + /* A NULL start produces an empty set. */ + if (PG_ARGISNULL(0)) + PG_RETURN_VOID(); + + start_lsn = PG_GETARG_LSN(0); + end_lsn = PG_ARGISNULL(1) ? start_lsn : PG_GETARG_LSN(1); + + if (start_lsn < XLOG_BLCKSZ) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not read WAL at LSN %X/%08X", + LSN_FORMAT_ARGS(start_lsn))); + + ValidateInputLSNs(start_lsn, &end_lsn); + point_lookup = start_lsn == end_lsn; + current_lsn = GetCurrentLSN(NULL); + + /* + * At a segment boundary, the current LSN can point just past the last + * retained WAL byte, before the next segment file has been created. In + * that case, return the preceding segment containing the available WAL. + */ + if (point_lookup && start_lsn == current_lsn && + XLogSegmentOffset(start_lsn, wal_segment_size) == 0) + XLByteToPrevSeg(start_lsn, first_segno, wal_segment_size); + else + XLByteToSeg(start_lsn, first_segno, wal_segment_size); + if (point_lookup) + last_segno = first_segno; + else + XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); + segments = GetWalTimeSegments(&nsegments); + + for (XLogSegNo segno = first_segno;; segno++) + { + Datum values[PG_GET_WAL_FILES_COLS]; + bool nulls[PG_GET_WAL_FILES_COLS] = {0}; + XLogRecPtr segment_start_lsn = segno * wal_segment_size; + XLogRecPtr segment_end_lsn = segment_start_lsn + wal_segment_size; + char fname[MAXFNAMELEN]; + + while (segment_index < nsegments && + segments[segment_index].segno < segno) + segment_index++; + + if (segment_index >= nsegments || + segments[segment_index].segno != segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the requested range is missing"), + errdetail("Segment " UINT64_FORMAT " is not present in pg_wal.", + (uint64) segno)); + + XLogFileName(fname, segments[segment_index].tli, segno, + wal_segment_size); + values[0] = CStringGetTextDatum(fname); + values[1] = LSNGetDatum(segment_start_lsn); + values[2] = LSNGetDatum(segment_end_lsn); + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, + values, nulls); + + if (segno == last_segno) + break; + + CHECK_FOR_INTERRUPTS(); + } + + pfree(segments); + PG_RETURN_VOID(); +#undef PG_GET_WAL_FILES_COLS +} + /* * Validate start and end LSNs coming from the function inputs. * @@ -525,7 +726,7 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) static void ValidateInputLSNs(XLogRecPtr start_lsn, XLogRecPtr *end_lsn) { - XLogRecPtr curr_lsn = GetCurrentLSN(); + XLogRecPtr curr_lsn = GetCurrentLSN(NULL); if (start_lsn > curr_lsn) ereport(ERROR, @@ -831,7 +1032,7 @@ Datum pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); if (start_lsn > end_lsn) ereport(ERROR, @@ -849,7 +1050,7 @@ Datum pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) { XLogRecPtr start_lsn = PG_GETARG_LSN(0); - XLogRecPtr end_lsn = GetCurrentLSN(); + XLogRecPtr end_lsn = GetCurrentLSN(NULL); bool stats_per_record = PG_GETARG_BOOL(1); if (start_lsn > end_lsn) diff --git a/contrib/pg_walinspect/pg_walinspect.control b/contrib/pg_walinspect/pg_walinspect.control index efa3cb2cfe5..5f574b865bb 100644 --- a/contrib/pg_walinspect/pg_walinspect.control +++ b/contrib/pg_walinspect/pg_walinspect.control @@ -1,5 +1,5 @@ # pg_walinspect extension comment = 'functions to inspect contents of PostgreSQL Write-Ahead Log' -default_version = '1.1' +default_version = '1.2' module_pathname = '$libdir/pg_walinspect' relocatable = true diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql index e35c4f3ed15..b935a067b3e 100644 --- a/contrib/pg_walinspect/sql/oldextversions.sql +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -27,10 +27,11 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); SELECT * FROM pg_get_wal_records_info_till_end_of_wal('FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_stats_till_end_of_wal('FFFFFFFF/FFFFFFFF'); --- Move to new version 1.1. +-- Test each upgrade step through the current version. ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; --- List what version 1.1 contains. +-- List what the current version contains. \dx+ pg_walinspect SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 1e64a22d29a..3176a17d855 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -51,6 +51,30 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn1', :'wal_lsn2'); +-- Return the retained WAL file and its complete segment boundaries. +SELECT wal_file = pg_walfile_name(segment_start_lsn) AS file_ok, + segment_start_lsn <= :'wal_lsn1'::pg_lsn AS start_ok, + segment_end_lsn > :'wal_lsn1'::pg_lsn AS end_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1); + +-- An end LSN at a segment boundary must not include the next segment. +WITH segment AS +( + SELECT * FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'::pg_lsn + 1) +) +SELECT count(*) = 1 AS one_file +FROM segment, + LATERAL pg_get_wal_files(segment_start_lsn, segment_end_lsn); + +SELECT count(*) = 1 AS equal_lsn_ok +FROM pg_get_wal_files(:'wal_lsn1', :'wal_lsn1'); +SELECT count(*) = 1 AS default_end_ok +FROM pg_get_wal_files(:'wal_lsn1'); +SELECT * FROM pg_get_wal_files(:'wal_lsn1'::pg_lsn + 1, :'wal_lsn1'); +SELECT count(*) = 1 AS future_end_ok +FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); +SELECT * FROM pg_get_wal_files('0/0', '0/1'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -110,6 +134,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files. GRANT pg_read_server_files TO regress_pg_walinspect; @@ -122,6 +148,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE pg_read_server_files FROM regress_pg_walinspect; @@ -134,6 +162,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes @@ -143,6 +173,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; @@ -152,6 +184,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) + FROM regress_pg_walinspect; -- =================================================================== -- Clean up diff --git a/contrib/pg_walinspect/t/001_timeline.pl b/contrib/pg_walinspect/t/001_timeline.pl new file mode 100644 index 00000000000..959fe485268 --- /dev/null +++ b/contrib/pg_walinspect/t/001_timeline.pl @@ -0,0 +1,74 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test that pg_get_wal_files() follows the current timeline's history across +# a promotion. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +sub test_timeline_history +{ + my ($suffix, $archive_mode) = @_; + my $primary = PostgreSQL::Test::Cluster->new("primary_$suffix"); + my $standby = PostgreSQL::Test::Cluster->new("standby_$suffix"); + + $primary->init(allows_streaming => 1); + $primary->append_conf('postgresql.conf', "autovacuum = off\n"); + if ($archive_mode) + { + $primary->append_conf( + 'postgresql.conf', + "archive_mode = on\narchive_command = 'true'\n"); + } + $primary->start; + $primary->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + my $boundary_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_lsn()'); + my $boundary_offset = $primary->safe_psql('postgres', + "SELECT file_offset FROM pg_walfile_name_offset('$boundary_lsn')"); + is($boundary_offset, '0', + "WAL is at a segment boundary with archive_mode=$archive_mode"); + my $boundary_files = $primary->safe_psql('postgres', + "SELECT count(*) FROM pg_get_wal_files('$boundary_lsn')"); + is($boundary_files, '1', + "point lookup works at a segment boundary with archive_mode=$archive_mode"); + + my $backup_name = "backup_$suffix"; + $primary->backup($backup_name); + $standby->init_from_backup($primary, $backup_name, has_streaming => 1); + $standby->append_conf('postgresql.conf', "wal_keep_size = '64MB'\n"); + $standby->start; + + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (1)'); + my $start_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); + $primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); + $primary->safe_psql('postgres', 'INSERT INTO test_table VALUES (2)'); + $primary->wait_for_catchup($standby); + + $standby->promote; + $standby->safe_psql('postgres', 'INSERT INTO test_table VALUES (3)'); + + my $result = $standby->safe_psql( + 'postgres', qq{ +SELECT count(DISTINCT left(wal_file, 8)) = 2, + min(left(wal_file, 8)) = '00000001', + max(left(wal_file, 8)) = '00000002' +FROM pg_get_wal_files('$start_lsn', pg_current_wal_lsn())}); + is($result, 't|t|t', + "WAL files span the timeline switch with archive_mode=$archive_mode"); + + $standby->stop; + $primary->stop; + return; +} + +test_timeline_history('archive_off', 0); +test_timeline_history('archive_on', 1); + +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 79c3ead40bc..2f708b32c60 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -34,9 +34,8 @@ </note> <tip> <para> - All of the <filename>pg_walinspect</filename> functions that show - information about records that fall within a certain LSN range are - permissive about accepting <replaceable>end_lsn</replaceable> + All of the <filename>pg_walinspect</filename> functions that accept an LSN + range are permissive about accepting <replaceable>end_lsn</replaceable> arguments that are after the server's current LSN. Using an <replaceable>end_lsn</replaceable> <quote>from the future</quote> will not raise an error. @@ -60,6 +59,42 @@ <title>General Functions</title> <variablelist> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-files"> + <term> + <function>pg_get_wal_files(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record</function> + </term> + + <listitem> + <para> + If <replaceable>end_lsn</replaceable> is specified and differs from + <replaceable>start_lsn</replaceable>, returns each retained WAL segment + file that intersects the half-open WAL range + [<replaceable>start_lsn</replaceable>, + <replaceable>end_lsn</replaceable>). If + <replaceable>end_lsn</replaceable> is omitted, is null, or equals + <replaceable>start_lsn</replaceable>, returns the segment containing + <replaceable>start_lsn</replaceable>. The + <structfield>wal_file</structfield> column contains the file name, while + <structfield>segment_start_lsn</structfield> and + <structfield>segment_end_lsn</structfield> contain the complete segment + boundaries rather than boundaries clipped to the requested range. + If a point lookup specifies the current WAL LSN and it lies exactly on + a segment boundary, the preceding segment is returned, since the next + segment might not have been created yet. This differs from + <function>pg_walfile_name</function>, which performs a pure calculation + that maps an LSN to a WAL file name without checking whether the file + exists, and therefore maps such a boundary LSN to the next segment. + </para> + + <para> + File names use the timeline valid for each segment in the current + timeline's history. This allows a returned range to cross a timeline + switch. The function reports an error if any required segment file is + not currently retained in <filename>pg_wal</filename>. + </para> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3bed561f7f2..46da9e8a15c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3430,6 +3430,7 @@ WalSummarizerData WalSummaryFile WalSummaryIO WalTimeSample +WalTimeSegment WalUsage WalWriteMethod WalWriteMethodOps -- 2.50.1 (Apple Git-155) =
v5-0002-pg_walinspect-add-function-to-locate-WAL-by-time.patch (application/octet-stream, 60.8K) download From a1dce53ea57647715e672c0b2543825a67669792 Mon Sep 17 00:00:00 2001 From: "Chao Li (Evan)" <lic@highgo.com> Date: Thu, 1 Oct 2026 06:52:59 +0800 Subject: [PATCH v5 2/2] pg_walinspect: add function to locate WAL by time Add pg_get_wal_location_at_time(), which uses timestamped WAL records to locate a useful WAL range around a requested wall-clock time. Use an approximate binary search over retained segments and allow callers to widen the requested interval when timestamp reordering makes the initial result insufficient. Move WAL record timestamp extraction from recovery into xlogreader so pg_walinspect can recognize the same timestamp-bearing record types without duplicating that logic. Author: Chao Li <lic@highgo.com> Reviewed-by: Surya Poondla <suryapoondla4@gmail.com> Discussion: https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com --- .../pg_walinspect/expected/oldextversions.out | 9 +- .../pg_walinspect/expected/pg_walinspect.out | 164 ++++++ contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.1--1.2.sql | 21 + contrib/pg_walinspect/pg_walinspect.c | 485 +++++++++++++++++- contrib/pg_walinspect/sql/pg_walinspect.sql | 121 +++++ contrib/pg_walinspect/t/002_time_search.pl | 98 ++++ doc/src/sgml/pgwalinspect.sgml | 139 +++++ src/backend/access/transam/xlogreader.c | 36 ++ src/backend/access/transam/xlogrecovery.c | 48 +- src/include/access/xlogreader.h | 3 + src/tools/pgindent/typedefs.list | 1 + 12 files changed, 1077 insertions(+), 49 deletions(-) create mode 100644 contrib/pg_walinspect/t/002_time_search.pl diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out index 311017d66ca..6228ad9ef39 100644 --- a/contrib/pg_walinspect/expected/oldextversions.out +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -51,15 +51,16 @@ ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; ALTER EXTENSION pg_walinspect UPDATE TO '1.2'; -- List what the current version contains. \dx+ pg_walinspect - Objects in extension "pg_walinspect" - Object description -------------------------------------------------------- + Objects in extension "pg_walinspect" + Object description +---------------------------------------------------------------------------------- function pg_get_wal_block_info(pg_lsn,pg_lsn,boolean) function pg_get_wal_files(pg_lsn,pg_lsn) + function pg_get_wal_location_at_time(timestamp with time zone,interval,interval) function pg_get_wal_record_info(pg_lsn) function pg_get_wal_records_info(pg_lsn,pg_lsn) function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) -(5 rows) +(6 rows) SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); pg_drop_replication_slot diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index 34864c65ec7..341bb543295 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -139,6 +139,145 @@ FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); ERROR: could not read WAL at LSN 0/00000000 -- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + start_ok | end_ok | lsn_ok +----------+--------+-------- + t | t | t +(1 row) + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + timestamped_ok +---------------- + t +(1 row) + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + start_ok | end_ok +----------+-------- + t | t +(1 row) + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + start_ok | current_end_ok | end_lsn_ok | lsn_ok | records_ok +----------+----------------+------------+--------+------------ + t | t | t | t | t +(1 row) + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; + defaults_ok +------------- + t +(1 row) + +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +ERROR: before interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +ERROR: after interval must be finite and greater than zero +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +ERROR: before interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +ERROR: after interval must not exceed one day +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +ERROR: could not locate a WAL time boundary +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); +ERROR: requested time follows the available WAL range +-- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset @@ -231,6 +370,13 @@ SELECT has_function_privilege('regress_pg_walinspect', f (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no + has_function_privilege +------------------------ + f +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no has_function_privilege @@ -268,6 +414,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -285,6 +438,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', @@ -315,6 +470,13 @@ SELECT has_function_privilege('regress_pg_walinspect', t (1 row) +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes + has_function_privilege +------------------------ + t +(1 row) + SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes has_function_privilege @@ -330,6 +492,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 6f8b4a6359a..79e94e33e4d 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -41,6 +41,7 @@ tests += { 'tap': { 'tests': [ 't/001_timeline.pl', + 't/002_time_search.pl', ], }, } diff --git a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql index 9cc9523b4dc..f72d99a3ceb 100644 --- a/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql +++ b/contrib/pg_walinspect/pg_walinspect--1.1--1.2.sql @@ -3,6 +3,27 @@ -- complain if script is sourced in psql, rather than via ALTER EXTENSION \echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.2'" to load this file. \quit +-- +-- pg_get_wal_location_at_time() +-- +CREATE FUNCTION pg_get_wal_location_at_time( + IN target_time timestamptz, + IN before interval DEFAULT '1 minute', + IN after interval DEFAULT '1 minute', + OUT start_timestamp timestamptz, + OUT start_lsn pg_lsn, + OUT end_timestamp timestamptz, + OUT end_lsn pg_lsn +) +AS 'MODULE_PATHNAME', 'pg_get_wal_location_at_time' +LANGUAGE C STRICT PARALLEL SAFE; + +REVOKE EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION + pg_get_wal_location_at_time(timestamptz, interval, interval) + TO pg_read_server_files; + -- -- pg_get_wal_files() -- diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index 85d76fe37c4..d57ac4f80ae 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -12,6 +12,8 @@ */ #include "postgres.h" +#include <sys/stat.h> + #include "access/htup_details.h" #include "access/timeline.h" #include "access/xlog.h" @@ -26,7 +28,9 @@ #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgrprotos.h" #include "utils/pg_lsn.h" +#include "utils/timestamp.h" #include "utils/tuplestore.h" /* @@ -41,6 +45,7 @@ PG_MODULE_MAGIC_EXT( PG_FUNCTION_INFO_V1(pg_get_wal_block_info); PG_FUNCTION_INFO_V1(pg_get_wal_files); +PG_FUNCTION_INFO_V1(pg_get_wal_location_at_time); PG_FUNCTION_INFO_V1(pg_get_wal_record_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info); PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); @@ -71,15 +76,52 @@ static void GetWalStats(FunctionCallInfo fcinfo, static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, bool show_data); +/* Selected WAL position and timestamp for one time boundary. */ +typedef struct WalTimeBoundary +{ + bool found; /* matching record was found */ + TimestampTz time; /* record's local event time */ + XLogRecPtr lsn; /* record's start LSN */ + int segment_index; /* index of the segment containing the record */ +} WalTimeBoundary; + /* Metadata for one retained segment in the current timeline's history. */ typedef struct WalTimeSegment { XLogSegNo segno; /* segment number, used for WAL ordering */ TimeLineID tli; /* timeline containing this segment */ + bool scanned; /* segment has already been decoded */ + bool has_timestamp; /* segment contains timestamped records */ + TimestampTz min_time; /* minimum record timestamp in segment */ + TimestampTz max_time; /* maximum record timestamp in segment */ + WalTimeBoundary lower; /* best lower anchor in segment */ + WalTimeBoundary upper; /* best upper anchor in segment */ } WalTimeSegment; static int wal_time_segment_cmp(const void *a, const void *b); -static WalTimeSegment *GetWalTimeSegments(int *nsegments); +static WalTimeSegment *GetWalTimeSegments(int *nsegments, + XLogRecPtr *current_lsn_out); +static int FindWalTimeCandidate(WalTimeSegment *segments, + int *search_segments, + int *nsearch_segments, + TimestampTz target_time, + bool search_left, + TimestampTz wanted_start, + TimestampTz wanted_end, + int *candidate_position, + int *nscanned); +static void ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, + TimestampTz wanted_end); +static void MergeWalTimeSegment(WalTimeSegment *segment, + WalTimeBoundary *lower, + WalTimeBoundary *upper); +static void ProbeLowerBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); +static void ProbeUpperBoundary(WalTimeBoundary *boundary, + TimestampTz time, XLogRecPtr lsn, + int segment_index); /* * Return the LSN up to which the server has WAL, and optionally its timeline. @@ -223,6 +265,46 @@ wal_time_segment_cmp(const void *a, const void *b) return 0; } +/* + * Probe a timestamped record at or before a requested time boundary. Among + * records examined so far, prefer the latest time, breaking ties in favor + * of the earlier WAL record. The caller only passes qualifying records, so + * a greater timestamp is closer to the requested boundary. + */ +static void +ProbeLowerBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time > boundary->time || + (time == boundary->time && lsn < boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + +/* + * Probe a timestamped record at or after a requested time boundary. Among + * records examined so far, prefer the earliest time, breaking ties in favor of + * the later WAL record. The caller only passes qualifying records, so a + * smaller timestamp is closer to the requested boundary. + */ +static void +ProbeUpperBoundary(WalTimeBoundary *boundary, TimestampTz time, + XLogRecPtr lsn, int segment_index) +{ + if (!boundary->found || time < boundary->time || + (time == boundary->time && lsn > boundary->lsn)) + { + boundary->found = true; + boundary->time = time; + boundary->lsn = lsn; + boundary->segment_index = segment_index; + } +} + /* * Get the WAL segments that form the history of the server's current * timeline. Other timelines may have files with the same segment number in @@ -230,7 +312,7 @@ wal_time_segment_cmp(const void *a, const void *b) * as read_local_xlog_page_no_wait() does. Return the segments in WAL order. */ static WalTimeSegment * -GetWalTimeSegments(int *nsegments) +GetWalTimeSegments(int *nsegments, XLogRecPtr *current_lsn_out) { WalTimeSegment *segments = NULL; DIR *dir; @@ -242,6 +324,8 @@ GetWalTimeSegments(int *nsegments) int count = 0; current_lsn = GetCurrentLSN(&current_tli); + if (current_lsn_out != NULL) + *current_lsn_out = current_lsn; history = readTimeLineHistory(current_tli); dir = AllocateDir(XLOGDIR); @@ -297,6 +381,140 @@ GetWalTimeSegments(int *nsegments) return segments; } +/* + * Approximately locate the segment containing target_time by binary search. + * Record timestamps are expected to generally increase with WAL position. + * Remove segments without timestamped records from the search candidates, + * since they provide no value with which to choose either half. + */ +static int +FindWalTimeCandidate(WalTimeSegment *segments, int *search_segments, + int *nsearch_segments, + TimestampTz target_time, bool search_left, + TimestampTz wanted_start, TimestampTz wanted_end, + int *candidate_position, int *nscanned) +{ + int left = 0; + int right = *nsearch_segments - 1; + int result; + + while (left <= right) + { + int middle = left + (right - left) / 2; + int segment_index = search_segments[middle]; + WalTimeSegment *segment = &segments[segment_index]; + + if (!segment->scanned) + { + ScanWalTimeSegment(segment, segment_index, wanted_start, wanted_end); + (*nscanned)++; + } + + if (!segment->has_timestamp) + { + memmove(&search_segments[middle], &search_segments[middle + 1], + (*nsearch_segments - middle - 1) * sizeof(int)); + (*nsearch_segments)--; + right--; + continue; + } + else if (target_time < segment->min_time) + right = middle - 1; + else if (target_time > segment->max_time) + left = middle + 1; + else + { + if (candidate_position != NULL) + *candidate_position = middle; + return search_segments[middle]; + } + } + + if (*nsearch_segments == 0) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not locate a WAL time boundary"), + errdetail("No retained WAL segment contains a timestamped record.")); + + if (search_left) + result = Max(right, 0); + else + result = Min(left, *nsearch_segments - 1); + + if (candidate_position != NULL) + *candidate_position = result; + return search_segments[result]; +} + +/* + * Inspect timestamped records beginning in one segment and save its time + * range and best boundary candidates. XLogReadRecord() may read the following + * segment to assemble a record crossing the boundary, but that record is + * attributed to the segment containing its starting LSN. + */ +static void +ScanWalTimeSegment(WalTimeSegment *segment, int segment_index, + TimestampTz wanted_start, TimestampTz wanted_end) +{ + XLogRecPtr seg_start = segment->segno * wal_segment_size; + XLogRecPtr seg_end = seg_start + wal_segment_size; + XLogReaderState *xlogreader; + + Assert(!segment->scanned); + segment->scanned = true; + xlogreader = InitXLogReaderState(seg_start); + + while (ReadNextXLogRecord(xlogreader)) + { + TimestampTz record_time; + + if (xlogreader->ReadRecPtr >= seg_end) + break; + + if (!GetXLogRecordTimestamp(xlogreader, &record_time)) + continue; + + if (!segment->has_timestamp) + { + segment->has_timestamp = true; + segment->min_time = segment->max_time = record_time; + } + else + { + segment->min_time = Min(segment->min_time, record_time); + segment->max_time = Max(segment->max_time, record_time); + } + + if (record_time <= wanted_start) + ProbeLowerBoundary(&segment->lower, record_time, + xlogreader->ReadRecPtr, segment_index); + + else if (record_time >= wanted_end) + ProbeUpperBoundary(&segment->upper, record_time, + xlogreader->ReadRecPtr, segment_index); + + CHECK_FOR_INTERRUPTS(); + } + + pfree(xlogreader->private_data); + XLogReaderFree(xlogreader); +} + +/* Merge one segment's boundary candidates into the result. */ +static void +MergeWalTimeSegment(WalTimeSegment *segment, WalTimeBoundary *lower, + WalTimeBoundary *upper) +{ + Assert(segment->scanned); + + if (lower != NULL && segment->lower.found) + ProbeLowerBoundary(lower, segment->lower.time, segment->lower.lsn, + segment->lower.segment_index); + if (upper != NULL && segment->upper.found) + ProbeUpperBoundary(upper, segment->upper.time, segment->upper.lsn, + segment->upper.segment_index); +} + /* * Output values that make up a row describing caller's WAL record. * @@ -623,6 +841,267 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) #undef PG_GET_WAL_RECORD_INFO_COLS } +/* + * Locate a WAL region around a wall-clock time range using timestamped WAL + * records. + * + * Locate each boundary by an approximate binary search over retained WAL + * segments, assuming that record timestamps generally increase with WAL + * position. Also examine one preceding segment for the lower boundary and + * one following segment for the upper boundary to allow for limited timestamp + * reordering. The lower boundary uses the latest timestamped record at or + * before the requested start time, while the upper boundary uses the earliest + * one at or after the requested end time. A future upper bound is first + * capped at the current time. If no upper timestamped record is available, + * use the current WAL position. On success, the boundaries form a forward + * WAL range, but are not necessarily the globally closest timestamped records + * to the requested bounds. + */ +Datum +pg_get_wal_location_at_time(PG_FUNCTION_ARGS) +{ +#define PG_GET_WAL_LOCATION_AT_TIME_COLS 4 + TimestampTz target_time = PG_GETARG_TIMESTAMPTZ(0); + Interval *before = PG_GETARG_INTERVAL_P(1); + Interval *after = PG_GETARG_INTERVAL_P(2); + Interval zero = {0}; + Interval one_day = {.day = 1}; + TimestampTz wanted_start; + TimestampTz wanted_end; + TimestampTz current_time; + XLogRecPtr current_lsn; + XLogSegNo current_segno; + WalTimeSegment *segments; + WalTimeBoundary lower = {0}; + WalTimeBoundary upper = {0}; + Datum values[PG_GET_WAL_LOCATION_AT_TIME_COLS]; + bool nulls[PG_GET_WAL_LOCATION_AT_TIME_COLS] = {0}; + TupleDesc tupdesc; + HeapTuple tuple; + int nsegments; + int *search_segments; + int nsearch_segments; + int lower_candidate; + int upper_candidate; + int lower_candidate_position; + int nupper_search_segments; + int lower_start; + int upper_end; + int upper_segment_index; + int nscanned = 0; + bool upper_bound_capped = false; + char lower_fname[MAXFNAMELEN]; + char upper_fname[MAXFNAMELEN]; + + if (TIMESTAMP_NOT_FINITE(target_time)) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("target time must be finite")); + if (INTERVAL_NOT_FINITE(before) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(before), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("before interval must not exceed one day")); + if (INTERVAL_NOT_FINITE(after) || + DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&zero))) <= 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must be finite and greater than zero")); + if (DatumGetInt32(DirectFunctionCall2(interval_cmp, + IntervalPGetDatum(after), + IntervalPGetDatum(&one_day))) > 0) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("after interval must not exceed one day")); + + /* The timestamp functions provide the normal overflow checks. */ + wanted_start = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_mi_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(before))); + wanted_end = DatumGetTimestampTz(DirectFunctionCall2(timestamptz_pl_interval, + TimestampTzGetDatum(target_time), + IntervalPGetDatum(after))); + current_time = GetCurrentTimestamp(); + + /* Cap a future upper bound at the time currently available. */ + if (wanted_end > current_time) + { + if (wanted_start >= current_time) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("requested time follows the available WAL range"), + errdetail("The requested lower bound is not earlier than the current time.")); + + wanted_end = current_time; + upper_bound_capped = true; + } + + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + segments = GetWalTimeSegments(&nsegments, &current_lsn); + XLByteToPrevSeg(current_lsn, current_segno, wal_segment_size); + + /* + * A capped upper bound may need the current WAL position as its endpoint. + * Detect a missing WAL tail before decoding any candidate segments. + */ + if (upper_bound_capped && + segments[nsegments - 1].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The current WAL LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(current_lsn), + (uint64) current_segno)); + + search_segments = palloc_array(int, nsegments); + for (int i = 0; i < nsegments; i++) + search_segments[i] = i; + nsearch_segments = nsegments; + lower_candidate = FindWalTimeCandidate(segments, search_segments, + &nsearch_segments, wanted_start, + true, wanted_start, wanted_end, + &lower_candidate_position, + &nscanned); + + /* Search for the later upper boundary from the lower candidate onward. */ + nupper_search_segments = nsearch_segments - lower_candidate_position; + upper_candidate = FindWalTimeCandidate(segments, + search_segments + lower_candidate_position, + &nupper_search_segments, wanted_end, + false, wanted_start, wanted_end, + NULL, + &nscanned); + XLogFileName(lower_fname, segments[lower_candidate].tli, + segments[lower_candidate].segno, wal_segment_size); + XLogFileName(upper_fname, segments[upper_candidate].tli, + segments[upper_candidate].segno, wal_segment_size); + ereport(DEBUG1, + errmsg_internal("WAL time search selected lower segment %s and upper segment %s from %d retained segments", + lower_fname, upper_fname, + nsegments)); + + /* + * Allow for limited timestamp reordering by examining one preceding + * segment for the lower boundary and one following segment for the upper + * boundary. Do not extend across a missing segment. + */ + lower_start = lower_candidate; + if (lower_start > 0 && + segments[lower_start - 1].segno + 1 == segments[lower_start].segno) + lower_start--; + upper_end = upper_candidate; + if (upper_end + 1 < nsegments && + segments[upper_end].segno + 1 == segments[upper_end + 1].segno) + upper_end++; + + for (int i = lower_start; i <= lower_candidate; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], &lower, NULL); + } + for (int i = upper_candidate; i <= upper_end; i++) + { + if (!segments[i].scanned) + { + ScanWalTimeSegment(&segments[i], i, wanted_start, wanted_end); + nscanned++; + } + MergeWalTimeSegment(&segments[i], NULL, &upper); + } + + ereport(DEBUG1, + errmsg_internal("WAL time search decoded %d of %d retained segments", + nscanned, nsegments)); + + if (!lower.found) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not locate a WAL time boundary"), + errdetail("The WAL search did not find a timestamped record at or before the requested lower bound.")); + + /* + * There may be no timestamped record after the requested upper bound on a + * quiet system. In that case, use the current WAL position so that the + * returned range still includes all WAL currently available. + */ + if (!upper.found) + { + upper.lsn = current_lsn; + + /* + * The last enumerated segment should contain current_lsn. Verify + * this because pg_wal can change while it is being enumerated; + * otherwise the continuity check below would not cover a missing WAL + * tail. + */ + upper_segment_index = nsegments - 1; + if (segments[upper_segment_index].segno != current_segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) current_segno)); + } + else + upper_segment_index = upper.segment_index; + + if (lower.lsn >= upper.lsn) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("could not find a valid WAL range for the requested time window"), + errdetail("The matching lower boundary does not precede the upper boundary in WAL.")); + + Assert(lower.segment_index >= 0 && lower.segment_index < nsegments); + Assert(upper_segment_index >= 0 && upper_segment_index < nsegments); + for (int i = lower.segment_index + 1; i <= upper_segment_index; i++) + { + /* + * The returned LSN range must be continuously readable from pg_wal; + * otherwise, functions such as pg_get_wal_records_info() would fail + * when scanning the range across a missing segment. + */ + if (segments[i - 1].segno + 1 != segments[i].segno) + ereport(ERROR, + errcode(ERRCODE_DATA_EXCEPTION), + errmsg("WAL segment needed for the time search is missing"), + errdetail("The start LSN is %X/%08X, the end LSN is %X/%08X, and segment " UINT64_FORMAT " is not present in pg_wal.", + LSN_FORMAT_ARGS(lower.lsn), + LSN_FORMAT_ARGS(upper.lsn), + (uint64) (segments[i - 1].segno + 1))); + } + + values[0] = TimestampTzGetDatum(lower.time); + values[1] = LSNGetDatum(lower.lsn); + if (upper.found) + values[2] = TimestampTzGetDatum(upper.time); + else + nulls[2] = true; + values[3] = LSNGetDatum(upper.lsn); + + pfree(segments); + tuple = heap_form_tuple(tupdesc, values, nulls); + PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); +#undef PG_GET_WAL_LOCATION_AT_TIME_COLS +} + /* * List the retained WAL segment files intersecting [start_lsn, end_lsn), or * the segment containing start_lsn when end_lsn is omitted or equal to it. @@ -676,7 +1155,7 @@ pg_get_wal_files(PG_FUNCTION_ARGS) last_segno = first_segno; else XLByteToPrevSeg(end_lsn, last_segno, wal_segment_size); - segments = GetWalTimeSegments(&nsegments); + segments = GetWalTimeSegments(&nsegments, NULL); for (XLogSegNo segno = first_segno;; segno++) { diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 3176a17d855..4b37a25fa41 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -75,6 +75,117 @@ SELECT count(*) = 1 AS future_end_ok FROM pg_get_wal_files(pg_current_wal_lsn(), 'FFFFFFFF/FFFFFFFF'); SELECT * FROM pg_get_wal_files('0/0', '0/1'); +-- =================================================================== +-- Tests for locating WAL by timestamps stored in WAL records +-- =================================================================== + +-- Put two WAL-logged transaction commits between the requested boundaries. +INSERT INTO sample_tbl VALUES (5, 5); +SELECT clock_timestamp() AS wal_time_lower \gset +INSERT INTO sample_tbl VALUES (6, 6); +INSERT INTO sample_tbl VALUES (10, 10); +SELECT clock_timestamp() AS wal_time_upper \gset +INSERT INTO sample_tbl VALUES (13, 13); + +SELECT start_timestamp <= :'wal_time_lower'::timestamptz - interval '1 microsecond' + AS start_ok, + end_timestamp >= :'wal_time_upper'::timestamptz AS end_ok, + start_lsn < end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 microsecond', + after => :'wal_time_upper'::timestamptz - + :'wal_time_lower'::timestamptz); + +-- A window containing one timestamped record is enclosed by outer anchors. +SELECT clock_timestamp() AS single_time_lower \gset +INSERT INTO sample_tbl VALUES (12, 12); +SELECT clock_timestamp() AS single_time_upper \gset +INSERT INTO sample_tbl VALUES (14, 14); +SELECT location.start_timestamp <= :'single_time_lower'::timestamptz + AS start_ok, + location.end_timestamp >= :'single_time_upper'::timestamptz + AS end_ok, + location.start_lsn < location.end_lsn AS lsn_ok +FROM pg_get_wal_location_at_time(:'single_time_lower', + interval '1 microsecond', + :'single_time_upper'::timestamptz - + :'single_time_lower'::timestamptz) AS location; + +-- Exercise a time window containing a COMMIT PREPARED record. Concurrent +-- activity can cause another timestamped record to be selected as the anchor. +INSERT INTO sample_tbl VALUES (11, 11); +BEGIN; +INSERT INTO sample_tbl VALUES (7, 7); +PREPARE TRANSACTION 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_target \gset +COMMIT PREPARED 'regress_pg_walinspect_time'; +SELECT clock_timestamp() AS prepared_time_upper \gset +INSERT INTO sample_tbl VALUES (15, 15); + +SELECT record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS timestamped_ok +FROM pg_get_wal_location_at_time(:'prepared_time_target', + :'prepared_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'prepared_time_upper'::timestamptz - + :'prepared_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.end_lsn); + +-- Exercise a time window containing restore points, allowing for other +-- timestamped WAL records generated concurrently. +SELECT pg_create_restore_point('regress_wal_time_lower') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_target \gset +SELECT pg_create_restore_point('regress_wal_time_upper') AS restore_lsn \gset +SELECT clock_timestamp() AS restore_time_upper \gset +INSERT INTO sample_tbl VALUES (8, 8); + +SELECT start_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS start_ok, + end_info.record_type IN ('COMMIT', 'COMMIT_PREPARED', 'ABORT', + 'ABORT_PREPARED', 'RESTORE_POINT') AS end_ok +FROM pg_get_wal_location_at_time(:'restore_time_target', + :'restore_time_target'::timestamptz - + :'wal_time_upper'::timestamptz, + :'restore_time_upper'::timestamptz - + :'restore_time_target'::timestamptz) AS location, + LATERAL pg_get_wal_record_info(location.start_lsn) AS start_info, + LATERAL pg_get_wal_record_info(location.end_lsn) AS end_info; + +-- A future upper bound falls back to the current WAL position. +SELECT clock_timestamp() AS current_time_target \gset +INSERT INTO sample_tbl VALUES (9, 9); +SELECT location.start_timestamp <= :'restore_time_upper'::timestamptz + AS start_ok, + location.end_timestamp IS NULL AS current_end_ok, + location.end_lsn <= pg_current_wal_flush_lsn() AS end_lsn_ok, + location.start_lsn < location.end_lsn AS lsn_ok, + EXISTS (SELECT FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'INSERT') + AS records_ok +FROM pg_get_wal_location_at_time(:'current_time_target', + :'current_time_target'::timestamptz - + :'restore_time_upper'::timestamptz, + interval '1 day') AS location; + +SELECT pg_get_function_arguments( + 'pg_get_wal_location_at_time(timestamptz, interval, interval)'::regprocedure) + LIKE '%before interval DEFAULT ''@ 1 min''::interval, after interval DEFAULT ''@ 1 min''::interval%' + AS defaults_ok; +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '0', interval '1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + interval '1 second', interval '0'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '-1 second'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + before => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(:'wal_time_lower', + after => interval '1 day 1 microsecond'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() - interval '100 years'); +SELECT * FROM pg_get_wal_location_at_time(clock_timestamp() + interval '100 years'); + -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== @@ -134,6 +245,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- no SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- no @@ -148,6 +261,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -162,6 +277,8 @@ GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; +GRANT EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + TO regress_pg_walinspect; GRANT EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) TO regress_pg_walinspect; @@ -173,6 +290,8 @@ SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes +SELECT has_function_privilege('regress_pg_walinspect', + 'pg_get_wal_location_at_time(timestamptz, interval, interval)', 'EXECUTE'); -- yes SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_files(pg_lsn, pg_lsn)', 'EXECUTE'); -- yes @@ -184,6 +303,8 @@ REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; +REVOKE EXECUTE ON FUNCTION pg_get_wal_location_at_time(timestamptz, interval, interval) + FROM regress_pg_walinspect; REVOKE EXECUTE ON FUNCTION pg_get_wal_files(pg_lsn, pg_lsn) FROM regress_pg_walinspect; diff --git a/contrib/pg_walinspect/t/002_time_search.pl b/contrib/pg_walinspect/t/002_time_search.pl new file mode 100644 index 00000000000..96a5c06eb76 --- /dev/null +++ b/contrib/pg_walinspect/t/002_time_search.pl @@ -0,0 +1,98 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test the bounded binary search for WAL time boundaries across segments. +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $node = PostgreSQL::Test::Cluster->new('time_search'); +$node->init(extra => ['--wal-segsize=1']); +$node->append_conf('postgresql.conf', "autovacuum = off\n"); +$node->start; +$node->safe_psql('postgres', + 'CREATE EXTENSION pg_walinspect; CREATE TABLE test_table (a int);'); +$node->safe_psql('postgres', 'SELECT pg_switch_wal()'); + +my ($lower_time, $upper_time); +for my $i (0 .. 8) +{ + $node->safe_psql('postgres', "INSERT INTO test_table VALUES ($i)"); + $lower_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 3; + $upper_time = $node->safe_psql('postgres', 'SELECT clock_timestamp()') + if $i == 5; + $node->safe_psql('postgres', 'SELECT pg_switch_wal()') if $i < 8; +} + +my $log_offset = -s $node->logfile; +my $result = $node->safe_psql( + 'postgres', qq{ +SET log_min_messages = debug1; +SELECT start_timestamp <= '$lower_time'::timestamptz - interval '1 microsecond', + end_timestamp >= '$upper_time'::timestamptz, + start_lsn < end_lsn, + pg_walfile_name(start_lsn) <> pg_walfile_name(end_lsn) +FROM pg_get_wal_location_at_time('$lower_time', + interval '1 microsecond', + '$upper_time'::timestamptz - + '$lower_time'::timestamptz)}); +is($result, 't|t|t|t', 'binary search finds outer boundaries across WAL segments'); + +my $log = slurp_file($node->logfile, $log_offset); +$log =~ /WAL time search decoded (\d+) of (\d+) retained segments/ + or die "could not find WAL time search statistics in server log"; +cmp_ok($1, '<', $2, 'binary search does not decode all retained WAL'); + +# A window containing no timestamped record is enclosed by the timestamped +# records immediately outside it. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS empty_window_start \gset +SELECT count(*) AS ignored FROM generate_series(1, 100000) \gset +SELECT clock_timestamp() AS empty_window_end \gset +INSERT INTO test_table VALUES (9); + +SELECT location.start_timestamp < :'empty_window_start'::timestamptz, + location.end_timestamp > :'empty_window_end'::timestamptz, + location.start_lsn < location.end_lsn +FROM pg_get_wal_location_at_time( + :'empty_window_start', + interval '1 microsecond', + :'empty_window_end'::timestamptz - + :'empty_window_start'::timestamptz) AS location;}); +is($result, 't|t|t', + 'outer anchors enclose a window with no timestamped record'); + +# With no later timestamped record, use the current WAL position as the upper +# boundary. The returned range must include WAL generated before the only +# commit record in the requested window. +$result = $node->safe_psql( + 'postgres', q{ +SELECT clock_timestamp() AS window_start \gset +BEGIN; +SELECT pg_current_wal_insert_lsn() AS delete_lsn \gset +DELETE FROM test_table WHERE a = 5; +COMMIT; +SELECT clock_timestamp() AS window_end \gset + +SELECT location.start_timestamp <= + :'window_start'::timestamptz - interval '1 microsecond', + location.end_timestamp IS NULL, + location.start_lsn < :'delete_lsn'::pg_lsn, + location.end_lsn > :'delete_lsn'::pg_lsn, + EXISTS (SELECT + FROM pg_get_wal_records_info(location.start_lsn, + location.end_lsn) + WHERE resource_manager = 'Heap' AND record_type = 'DELETE') +FROM pg_get_wal_location_at_time( + :'window_start', + interval '1 microsecond', + :'window_end'::timestamptz - :'window_start'::timestamptz) AS location;}); +is($result, 't|t|t|t|t', + 'current WAL fallback includes changes preceding the only commit in the window'); + +$node->stop; +done_testing(); diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 2f708b32c60..daa5b7ed0b2 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -95,6 +95,145 @@ </listitem> </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-location-at-time"> + <term> + <function>pg_get_wal_location_at_time(target_time timestamptz, before interval DEFAULT '1 minute', after interval DEFAULT '1 minute') returns record</function> + </term> + + <listitem> + <para> + Locates a nearby WAL region using records containing local event + timestamps as historical time anchors. The requested time range extends + from <replaceable>target_time</replaceable> minus + <replaceable>before</replaceable> through + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable>. It returns + <structfield>start_timestamp</structfield>, + <structfield>start_lsn</structfield>, + <structfield>end_timestamp</structfield>, and + <structfield>end_lsn</structfield>. The effective upper bound is the + earlier of <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> and the current time. On successful + return, the following relationships hold: +<programlisting> +start_timestamp &lt;= target_time - before +end_timestamp &gt;= effective upper bound, or is null +start_lsn &lt; end_lsn +</programlisting> + The start boundary is the latest timestamped record found at or before + the requested start time, and the end boundary is the earliest + timestamped record found at or after the requested end time. If + <replaceable>target_time</replaceable> plus + <replaceable>after</replaceable> is later than the current time, the + requested upper bound is first capped at the current time. If no + timestamped record is available at or after the upper bound, + <structfield>end_lsn</structfield> is the current flush or replay + position and <structfield>end_timestamp</structfield> is null. Both + interval arguments must be greater than zero and not greater than one + day. + </para> + + <para> + Most WAL records do not contain a wall-clock timestamp. Historical + anchors can be transaction <literal>COMMIT</literal>, + <literal>COMMIT PREPARED</literal>, <literal>ABORT</literal>, + or <literal>ABORT PREPARED</literal> records, as well as restore point + records. These are the WAL record types from which PostgreSQL recovery + can extract timestamps. Their timestamps describe the corresponding + event; they do not assign timestamps to heap, index, or other records. + </para> + + <para> + In particular, a commit timestamp describes when a transaction + committed, not when its earlier changes generated WAL. For example, a + transaction might update a row at 14:00 and commit at 15:00; the heap + WAL was generated around 14:00, while the commit record is timestamped + 15:00. Using timestamped records outside the requested time window as + boundaries helps include the WAL generated by transactions that commit + within the window. However, the returned LSNs do not identify the exact + generation time of heap, index, or other WAL records, and a transaction + can have generated WAL before the selected start boundary. On a cluster + with little timestamped activity, the returned timestamps can be far + from the requested bounds. + </para> + + <para> + The function assumes that WAL record timestamps generally increase with + WAL position and uses an approximate binary search to locate each time + boundary. WAL record timestamps are not guaranteed to follow WAL order, + for example because of concurrent activity, clock adjustments, or + delayed WAL insertion. Such timestamp reordering can cause relevant + records to fall outside the returned range. Therefore, the returned + records are not guaranteed to be the globally closest timestamped + records to either requested time bound. If the returned range does not + contain the records of interest, the user can widen the search range and + repeat the search. If the matching lower boundary follows the upper + boundary in WAL, the function reports an error rather than returning a + range with the LSNs reversed. + </para> + + <para> + Only WAL that is currently retained in <filename>pg_wal</filename> can + be searched. The function reports an error when a lower boundary cannot + be found by the bounded search or when the resulting range would cross a + missing WAL segment. When an upper timestamped boundary cannot be + found, the current WAL position is returned instead, as described above. + </para> + + <para> + For example, a DBA can locate WAL around activity near a given time and + then inspect the returned range: +<programlisting> +SELECT * +FROM pg_get_wal_location_at_time( + '2026-08-01 14:35:00+08', + before =&gt; '10 minutes', + after =&gt; '10 minutes'); + +SELECT * +FROM pg_get_wal_files('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); + +SELECT * +FROM pg_get_wal_records_info('<replaceable>start_lsn</replaceable>', + '<replaceable>end_lsn</replaceable>'); +</programlisting> + The corresponding retained WAL files can be obtained with + <function>pg_get_wal_files</function>. To widen the search range, + increase <replaceable>before</replaceable> if the relevant activity + might have occurred earlier, or increase + <replaceable>after</replaceable> if it might have occurred later, and + repeat the search. Failure to find a record in the first returned range + does not necessarily mean that the record is absent from retained WAL. + </para> + + <tip> + <para> + When <structfield>end_timestamp</structfield> is not null, + <structfield>end_lsn</structfield> points to the start of the end-anchor + record. Therefore, passing the returned LSN range directly to + <function>pg_get_wal_records_info</function> does not include that + record. For example, if the end anchor happens to be a transaction + commit, the range can contain the transaction's earlier changes but not + its commit record. + </para> + + <para> + To include the end-anchor record, obtain its ending LSN with + <function>pg_get_wal_record_info</function> and use that as the range + endpoint: +<programlisting> +SELECT * +FROM pg_get_wal_records_info( + '<replaceable>start_lsn</replaceable>', + (SELECT end_lsn + FROM pg_get_wal_record_info('<replaceable>end_lsn</replaceable>'))); +</programlisting> + </para> + </tip> + </listitem> + </varlistentry> + <varlistentry id="pgwalinspect-funcs-pg-get-wal-record-info"> <term> <function>pg_get_wal_record_info(in_lsn pg_lsn) returns record</function> diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c index 1db5ca48c3f..b91458b9636 100644 --- a/src/backend/access/transam/xlogreader.c +++ b/src/backend/access/transam/xlogreader.c @@ -26,6 +26,7 @@ #endif #include "access/transam.h" +#include "access/xact.h" #include "access/xlog_internal.h" #include "access/xlogreader.h" #include "access/xlogrecord.h" @@ -72,6 +73,41 @@ static void XLogReaderFreeZstdContext(void *arg); */ #define DEFAULT_DECODE_BUFFER_SIZE (64 * 1024) +/* + * Extract timestamp from WAL record. + * + * If the record contains a timestamp, returns true, and saves the timestamp + * in *recordXtime. If the record type has no timestamp, returns false. + * Currently, only transaction commit/abort records and restore points contain + * timestamps. + */ +bool +GetXLogRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) +{ + uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; + uint8 xact_info = info & XLOG_XACT_OPMASK; + uint8 rmid = XLogRecGetRmid(record); + + if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) + { + *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || + xact_info == XLOG_XACT_COMMIT_PREPARED)) + { + *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; + return true; + } + if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || + xact_info == XLOG_XACT_ABORT_PREPARED)) + { + *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; + return true; + } + return false; +} + /* * Construct a string in state->errormsg_buf explaining what's wrong with * the current record being read. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 54aaec9529f..06a5b10d092 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -356,7 +356,6 @@ static void xlog_outrec(StringInfo buf, XLogReaderState *record); static void xlog_block_info(StringInfo buf, XLogReaderState *record); static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, TimeLineID replayTLI); -static bool getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime); static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); @@ -2405,41 +2404,6 @@ checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI, TimeLineID prevTLI, } -/* - * Extract timestamp from WAL record. - * - * If the record contains a timestamp, returns true, and saves the timestamp - * in *recordXtime. If the record type has no timestamp, returns false. - * Currently, only transaction commit/abort records and restore points contain - * timestamps. - */ -static bool -getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime) -{ - uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK; - uint8 xact_info = info & XLOG_XACT_OPMASK; - uint8 rmid = XLogRecGetRmid(record); - - if (rmid == RM_XLOG_ID && info == XLOG_RESTORE_POINT) - { - *recordXtime = ((xl_restore_point *) XLogRecGetData(record))->rp_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_COMMIT || - xact_info == XLOG_XACT_COMMIT_PREPARED)) - { - *recordXtime = ((xl_xact_commit *) XLogRecGetData(record))->xact_time; - return true; - } - if (rmid == RM_XACT_ID && (xact_info == XLOG_XACT_ABORT || - xact_info == XLOG_XACT_ABORT_PREPARED)) - { - *recordXtime = ((xl_xact_abort *) XLogRecGetData(record))->xact_time; - return true; - } - return false; -} - /* * Checks whether the current buffer page and backup page stored in the * WAL record are consistent or not. Before comparing the two pages, a @@ -2663,10 +2627,10 @@ recoveryStopsBefore(XLogReaderState *record) /* * Note: we must fetch recordXtime regardless of recoveryTarget setting. - * We don't expect getRecordTimestamp ever to fail, since we already know - * this is a commit or abort record; but test its result anyway. + * We don't expect GetXLogRecordTimestamp ever to fail, since we already + * know this is a commit or abort record; but test its result anyway. */ - if (getRecordTimestamp(record, &recordXtime) && + if (GetXLogRecordTimestamp(record, &recordXtime) && recoveryTarget == RECOVERY_TARGET_TIME) { /* @@ -2747,7 +2711,7 @@ recoveryStopsAfter(XLogReaderState *record) recoveryStopAfter = true; recoveryStopXid = InvalidTransactionId; recoveryStopLSN = InvalidXLogRecPtr; - (void) getRecordTimestamp(record, &recoveryStopTime); + (void) GetXLogRecordTimestamp(record, &recoveryStopTime); strlcpy(recoveryStopName, recordRestorePointData->rp_name, MAXFNAMELEN); ereport(LOG, @@ -2787,7 +2751,7 @@ recoveryStopsAfter(XLogReaderState *record) TransactionId recordXid; /* Update the last applied transaction timestamp */ - if (getRecordTimestamp(record, &recordXtime)) + if (GetXLogRecordTimestamp(record, &recordXtime)) SetLatestXTime(recordXtime); /* Extract the XID of the committed/aborted transaction */ @@ -3006,7 +2970,7 @@ recoveryApplyDelay(XLogReaderState *record) xact_info != XLOG_XACT_COMMIT_PREPARED) return false; - if (!getRecordTimestamp(record, &xtime)) + if (!GetXLogRecordTimestamp(record, &xtime)) return false; delayUntil = TimestampTzPlusMilliseconds(xtime, recovery_min_apply_delay); diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h index e48a0db9ce9..43682a28272 100644 --- a/src/include/access/xlogreader.h +++ b/src/include/access/xlogreader.h @@ -39,6 +39,7 @@ #endif #include "access/xlogrecord.h" +#include "datatype/timestamp.h" #include "storage/buf.h" /* WALOpenSegment represents a WAL segment being read. */ @@ -462,5 +463,7 @@ extern bool XLogRecGetBlockTagExtended(XLogReaderState *record, uint8 block_id, RelFileLocator *rlocator, ForkNumber *forknum, BlockNumber *blknum, Buffer *prefetch_buffer); +extern bool GetXLogRecordTimestamp(XLogReaderState *record, + TimestampTz *recordXtime); #endif /* XLOGREADER_H */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 46da9e8a15c..8e555337a13 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3429,6 +3429,7 @@ WalSndState WalSummarizerData WalSummaryFile WalSummaryIO +WalTimeBoundary WalTimeSample WalTimeSegment WalUsage -- 2.50.1 (Apple Git-155) =
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-10-12 04:11  surya poondla <suryapoondla4@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  0 siblings, 0 replies; 12+ messages in thread

From: surya poondla @ 2026-10-12 04:11 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Thank you, Chao.

v5 addresses all of my comments. The doc and errcode changes look good. The
regression and TAP tests pass locally on master (f884f359f5a), CFBot is
green, and pgindent reports no changes for the C files.

The changes look good for a Committer.

Regards,
Surya Poondla

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-11-17 17:17  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  1 sibling, 1 reply; 12+ messages in thread

From: Bharath Rupireddy @ 2026-11-17 17:17 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Hi,

On Thu, Sep 17, 2026 at 9:06 PM Chao Li <li.evan.chao@gmail.com> wrote:
>
> I'd like to propose adding two new SQL functions to pg_walinspect.

Thanks, and sorry for the delay.

> During HOW2026 in Jinan this April, a DBA talked to me that PG lacks an easy way to locate WAL files by time.

I think using the existing pg_walinspect functions, one can look up
the times using the transaction commit/abort time from the description
column, starting from the last checkpoint's redo LSN till the current
flush LSN. Isn't this helpful? It takes a bit of time to do so, but
that should be fine.

> He described a scenario where somebody had done something wrong in a production database, for example accidentally deleting important data. The administrator knew that the incident had happened at around 1am the last night and wanted to confirm what had happened by analyzing WAL.

Isn't one expected to set up *some* kind of auditing or delayed
streaming standby or such in production if worried about such
accidental deletes/drops?

> The problem was that, with thousands of WAL files in pg_wal, where should he start?

WAL files are archived and renamed/reused/removed, so all the
thousands of files, one of which could be the interesting file
containing the accidental WAL records, are not guaranteed to be
available in the pg_wal dir anyway. 1000 WAL files come to around
16GB, and I'm not sure if anyone really prefers to delay checkpoints
beyond that point (max_wal_size is also usually set up to trigger the
checkpoint). If needed, one can use pg_waldump and some scripts to do
the analysis independently of the database instance.

I'm not against enhancing pg_walinspect for more use cases. However,
I'm a bit worried about complicating it for this. Unless I'm missing
something here, the reported issue could have been solved with a bit
of SQL over the existing functions and/or with some scripting with
pg_waldump over pg_wal dir or the archived WAL.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: pg_walinspect: add functions to locate and list WAL by time and LSN
@ 2026-11-18 03:16  Chao Li <li.evan.chao@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 0 replies; 12+ messages in thread

From: Chao Li @ 2026-11-18 03:16 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>



> On Nov 18, 2026, at 01:17, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:
> 
> Hi,
> 
> On Thu, Sep 17, 2026 at 9:06 PM Chao Li <li.evan.chao@gmail.com> wrote:
>> 
>> I'd like to propose adding two new SQL functions to pg_walinspect.
> 
> Thanks, and sorry for the delay.

Thanks a lot for your feedback.

> 
>> During HOW2026 in Jinan this April, a DBA talked to me that PG lacks an easy way to locate WAL files by time.
> 
> I think using the existing pg_walinspect functions, one can look up
> the times using the transaction commit/abort time from the description
> column, starting from the last checkpoint's redo LSN till the current
> flush LSN. Isn't this helpful? It takes a bit of time to do so, but
> that should be fine.

I'm afraid that's not easy. As I implemented in the patch, I even chose to use a binary search to find the start and end LSN boundaries for a time range. I don't think this can be done easily with just a few SQL statements, especially for ordinary DBAs. I've heard from many DBAs who have requested such a function. Surya also agreed that this is a real use case.

> 
>> He described a scenario where somebody had done something wrong in a production database, for example accidentally deleting important data. The administrator knew that the incident had happened at around 1am the last night and wanted to confirm what had happened by analyzing WAL.
> 
> Isn't one expected to set up *some* kind of auditing or delayed
> streaming standby or such in production if worried about such
> accidental deletes/drops?

We cannot imagine all users' operational models. I started talking to DBAs last year. My main takeaway is that real-world use cases are usually far more complicated than what I can imagine as a hacker writing PostgreSQL code. I didn't come up with this feature myself; some DBAs requested it.

> 
>> The problem was that, with thousands of WAL files in pg_wal, where should he start?
> 
> WAL files are archived and renamed/reused/removed, so all the
> thousands of files, one of which could be the interesting file
> containing the accidental WAL records, are not guaranteed to be
> available in the pg_wal dir anyway. 1000 WAL files come to around
> 16GB, and I'm not sure if anyone really prefers to delay checkpoints
> beyond that point (max_wal_size is also usually set up to trigger the
> checkpoint). If needed, one can use pg_waldump and some scripts to do
> the analysis independently of the database instance.

I guess users would be more likely to set wal_keep_size to a large value in order to retain more WAL files.

Yes, users can write scripts to do whatever kind of analysis. But here, users want a simple, native PG method. This patch is trying to address a user requirement and make DBAs' lives a bit easier.

Also, this patch doesn't touch the core, it just adds two new functions to the pg_walinspect extension.

> 
> I'm not against enhancing pg_walinspect for more use cases. However,
> I'm a bit worried about complicating it for this. Unless I'm missing
> something here, the reported issue could have been solved with a bit
> of SQL over the existing functions and/or with some scripting with
> pg_waldump over pg_wal dir or the archived WAL.
> 

I don't feel this patch adds significant complexity to pg_walinspect. This feature doesn't change the existing logic of pg_walinspect, it only adds two new functions. And as explained above, what the new functions do is not as easy to achieve with just a bit of SQL. Whenever possible, we should try to improve the user experience.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/










^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2026-11-18 03:16 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-09-18 04:05 pg_walinspect: add functions to locate and list WAL by time and LSN Chao Li <li.evan.chao@gmail.com>
2026-09-29 22:31 ` surya poondla <suryapoondla4@gmail.com>
2026-10-01 00:16   ` Chao Li <li.evan.chao@gmail.com>
2026-10-06 23:11     ` surya poondla <suryapoondla4@gmail.com>
2026-10-08 06:13       ` Chao Li <li.evan.chao@gmail.com>
2026-10-08 22:36         ` surya poondla <suryapoondla4@gmail.com>
2026-10-09 01:28           ` Chao Li <li.evan.chao@gmail.com>
2026-10-09 23:18             ` surya poondla <suryapoondla4@gmail.com>
2026-10-10 03:40               ` Chao Li <li.evan.chao@gmail.com>
2026-10-12 04:11                 ` surya poondla <suryapoondla4@gmail.com>
2026-11-17 17:17 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-11-18 03:16   ` Chao Li <li.evan.chao@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox