pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Return pg_control from pg_backup_stop().
64+ messages / 15 participants
[nested] [flat]

Commits from this thread:
  2026-10-29 c97931c16d 9c299e2924 4f857f399e 59bc83d415 38ecdfdc1f 4b87b605bb  Work around failure with SPARC in base backup code
  2026-10-27 4a92c8c29b e2c633b43b 8cddd5fc8f 4aeefb26d6 5dc3cfbb90 316a870654  Avoid torn copy of the control file in base backups
  2026-10-27 0cfd56254a d2c1e18de9  Extend sendFileWithContent() to handle custom content length in basebackup.c
* Return pg_control from pg_backup_stop().
@ 2024-05-17 02:46  David Steele <david@pgmasters.net>
  0 siblings, 2 replies; 64+ messages in thread

From: David Steele @ 2024-05-17 02:46 UTC (permalink / raw)
  To: pgsql-hackers

Hackers,

This is greatly simplified implementation of the patch proposed in [1] 
and hopefully it addresses the concerns expressed there. Since the 
implementation is quite different it seemed like a new thread was 
appropriate, especially since the old thread title would be very 
misleading regarding the new functionality.

The basic idea is to harden recovery by returning a copy of pg_control 
from pg_backup_stop() that has a flag set to prevent recovery if the 
backup_label file is missing. Instead of backup software copying 
pg_control from PGDATA, it stores an updated version that is returned 
from pg_backup_stop(). This is better for the following reasons:

* The user can no longer remove backup_label and get what looks like a 
successful recovery (while almost certainly causing corruption). If 
backup_label is removed the cluster will not start. The user may try 
pg_resetwal, but that tool makes it pretty clear that corruption will 
result from its use.

* We don't need to worry about backup software seeing a torn copy of 
pg_control, since Postgres can safely read it out of memory and provide 
a valid copy via pg_backup_stop(). This solves torn reads without 
needing to write pg_control via a temp file, which may affect 
performance on a standby.

* For backup from standby, we no longer need to instruct the backup 
software to copy pg_control last. In fact the backup software should not 
copy pg_control from PGDATA at all.

These changes have no impact on current backup software and they are 
free to use the pg_control available from pg_stop_backup() or continue 
to use pg_control from PGDATA. Of course they will miss the benefits of 
getting a consistent copy of pg_control and the backup_label checking, 
but will be no worse off than before.

I'll register this in the July CF.

Regards,
-David

[1] 
https://www.postgresql.org/message-id/2daf8adc-8db7-4204-a7f2-a7e94e2bfa4b@pgmasters.net

Attachments:

pgcontrol-from-backupstop-v1.patch (text/plain, 22.1K) download From 531872dcdb09f5d2f89af44a3c04c9d2d6da89be Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 17 May 2024 02:42:35 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 18 +++++- doc/src/sgml/func.sgml | 10 ++- src/backend/access/transam/xlog.c | 19 ++++++ src/backend/access/transam/xlogfuncs.c | 17 ++++-- src/backend/access/transam/xlogrecovery.c | 10 ++- src/backend/backup/basebackup.c | 19 +++--- src/backend/catalog/system_functions.sql | 2 +- src/backend/utils/misc/pg_controldata.c | 7 ++- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlogbackup.h | 8 +++ src/include/catalog/pg_control.h | 4 ++ src/include/catalog/pg_proc.dat | 10 +-- src/test/recovery/t/002_archiving.pl | 20 ++++++ src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 16 files changed, 182 insertions(+), 33 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 91da3c26ba..1b87a05dc5 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1010,9 +1010,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1084,7 +1087,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 17c44bc338..88f942af44 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27697,6 +27697,11 @@ SELECT currval(pg_get_serial_sequence('sometable', 'id')); <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> @@ -28355,8 +28360,9 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 The result of the function is a single record. The <parameter>lsn</parameter> column holds the backup's ending write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be + column returns the contents of the backup label file, the third column + returns the contents of the tablespace map file, and the fourth column + returns the contents of pg_control. These must be stored as part of the backup and are required as part of the restore process. </para> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index c3fd9c1eae..7d7c3d727e 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9048,11 +9048,30 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) int seconds_before_warning; int waits = 0; bool reported_waiting = false; + ControlFileData *controlFileCopy = (ControlFileData *)state->controlFile; Assert(state != NULL); backup_stopped_in_recovery = RecoveryInProgress(); + /* + * Create a copy of control data and update it to require a backup label + * for recovery. Also recalculate the CRC. + */ + memset( + state->controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupLabelRequired = true; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 92bdb17ed5..2576cacf22 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -112,9 +112,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains represents a consistent copy of pg_control that also has a safeguard + * against being used without backup_label. It is the caller's responsibility + * to write the backup_label, pg_control, and tablespace_map files in the data + * folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -122,12 +124,13 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -149,9 +152,15 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 29c5bec084..0940e4730e 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -703,7 +703,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to succeed"))); /* * If tablespace_map file is present without backup_label file, there @@ -976,6 +983,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 9a2bf59e84..ae0f83416a 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -324,7 +324,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; @@ -348,15 +347,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid, ib); + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_FILE_SIZE, &manifest); } else { @@ -390,9 +388,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, } } - basebackup_progress_wait_wal_archive(&state); - do_pg_backup_stop(backup_state, !opt->nowait); - endptr = backup_state->stoppoint; endtli = backup_state->stoptli; diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index ae099e328c..d76d3c608f 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 98c932dc7b..9eaf3f8b9f 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93e0837947..2114488dc0 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -283,6 +283,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index e9dcb5a6d8..7056752cff 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -875,6 +875,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 8449ae78ef..7c27d47110 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -722,6 +722,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h index c30d4a5991..93a0e5c5cc 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" /* Structure to hold backup state. */ @@ -35,6 +36,13 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. backupLabelRequired + * has been set so backup_label will be required for recovery to start. + */ + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; } BackupState; extern char *build_backup_content(BackupState *state, diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index a00606ffcd..71a9beb05a 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -163,12 +163,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to succeed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 6a5476d3c4..6cb87fb744 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6463,8 +6463,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', @@ -11981,9 +11981,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index a2e012e42d..653546a6c0 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 61d23187e0..bd3a99960f 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2024-10-02 07:15  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgmasters.net>
  1 sibling, 0 replies; 64+ messages in thread

From: Michael Paquier @ 2024-10-02 07:15 UTC (permalink / raw)
  To: pgsql-hackers

On Fri, May 17, 2024 at 12:46:49PM +1000, David Steele wrote:
> This is greatly simplified implementation of the patch proposed in [1] and
> hopefully it addresses the concerns expressed there. Since the
> implementation is quite different it seemed like a new thread was
> appropriate, especially since the old thread title would be very misleading
> regarding the new functionality.

-        /* No backup_label file has been found if we are here. */
+        /*
+         * No backup_label file has been found if we are here. Error if the
+         * control file requires backup_label.
+         */
+        if (ControlFile->backupLabelRequired)
+            ereport(FATAL,
+                    (errmsg("could not find backup_label required for recovery"),
+                     errhint("backup_label must be present for recovery to succeed")));

I thought that this had some similarities with my last fight in this
area, where xlogrecovery.c would fail hard if there was a backup_label
file but no signal files, but nope that's not the case:
https://www.postgresql.org/message-id/flat/ZArVOMifjzE7f8W7%40paquier.xyz

> The basic idea is to harden recovery by returning a copy of pg_control from
> pg_backup_stop() that has a flag set to prevent recovery if the backup_label
> file is missing. Instead of backup software copying pg_control from PGDATA,
> it stores an updated version that is returned from pg_backup_stop().

Hmm, okay.  There is also a slight impact for BASE_BACKUP, requiring
basebackup_progress_wait_wal_archive() and do_pg_backup_stop() to be
called earlier when sending the main data directory which is the last
one in the list of tablespaces.  As far as I can see, this does not
change the logic because do_pg_backup_stop() does not touch the
control file, but it seems to me that we'd rather keep these two
calls as they are now, and send the control file once we are out of
the for loop that processes all the tablespaces.  That seems slightly
cleaner to me, still I agree that both are the same things.

Anyway, finishing do_pg_backup_stop() and then sending the control
file is just a consequence of the implementation choice driven by the
output required for the SQL function so as this is stored in the
backup state to get it down to pg_backup_stop() in xlogfuncs.c, so we
could also take one step back and forget about the SQL function,
setting only the new flag when sending a BASE_BACKUP, or just not use
the backupState to store this data as the exercise involves forcing
one boolean and recalculate a CRC32.

> * We don't need to worry about backup software seeing a torn copy of
> pg_control, since Postgres can safely read it out of memory and provide a
> valid copy via pg_backup_stop(). This solves torn reads without needing to
> write pg_control via a temp file, which may affect performance on a standby.

We're talking about a 8kB file which has a size of 512B
(PG_CONTROL_MAX_SAFE_SIZE) to avoid such issues.  So I'm not sure to
see your point here?

> * For backup from standby, we no longer need to instruct the backup software
> to copy pg_control last. In fact the backup software should not copy
> pg_control from PGDATA at all.

Yep.  Not from PGDATA, but from the function.

> These changes have no impact on current backup software and they are free to
> use the pg_control available from pg_stop_backup() or continue to use
> pg_control from PGDATA. Of course they will miss the benefits of getting a
> consistent copy of pg_control and the backup_label checking, but will be no
> worse off than before.

There is a large comment block in do_pg_backup_stop() around
backup_stopped_in_recovery.  Perhaps it should be improved based on
this patch.

The main concern that I have over this patch is: who is actually going
to use this extension of the SQL stop function?  Perhaps existing
backup solutions are good enough risk vs reward is not worth it?  The
label_file and the tablespace map are text, this is a series of bytes
that has no visibility for the end-user unless checked on the
client-side.  This adds a new step where backups would need to copy
the control file to the data folder.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2024-10-02 09:03  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgmasters.net>
  1 sibling, 1 reply; 64+ messages in thread

From: David Steele @ 2024-10-02 09:03 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 10/2/24 10:11, Michael Paquier wrote:
> On Fri, May 17, 2024 at 12:46:49PM +1000, David Steele wrote:
> 
>> The basic idea is to harden recovery by returning a copy of pg_control from
>> pg_backup_stop() that has a flag set to prevent recovery if the backup_label
>> file is missing. Instead of backup software copying pg_control from PGDATA,
>> it stores an updated version that is returned from pg_backup_stop().
> 
> Hmm, okay.  There is also a slight impact for BASE_BACKUP, requiring
> basebackup_progress_wait_wal_archive() and do_pg_backup_stop() to be
> called earlier when sending the main data directory which is the last
> one in the list of tablespaces.  As far as I can see, this does not
> change the logic because do_pg_backup_stop() does not touch the
> control file, but it seems to me that we'd rather keep these two
> calls as they are now, and send the control file once we are out of
> the for loop that processes all the tablespaces.  That seems slightly
> cleaner to me, still I agree that both are the same things.

Sending pg_control later results in even more code churn because of how 
tar files are terminated. I've updated it that way in v2 so you can see 
what I mean. I don't have a strong preference, though, so if you prefer 
the implementation in v2 then that's fine with me.

> Anyway, finishing do_pg_backup_stop() and then sending the control
> file is just a consequence of the implementation choice driven by the
> output required for the SQL function so as this is stored in the
> backup state to get it down to pg_backup_stop() in xlogfuncs.c, so we
> could also take one step back and forget about the SQL function,
> setting only the new flag when sending a BASE_BACKUP, or just not use
> the backupState to store this data as the exercise involves forcing
> one boolean and recalculate a CRC32.

I can definitely see us making other updates to pg_control so I would 
rather keep this logic centralized, even though it is not too 
complicated at this point. Still, even 8 lines of code (as it is now) 
seems better not to duplicate.

>> * We don't need to worry about backup software seeing a torn copy of
>> pg_control, since Postgres can safely read it out of memory and provide a
>> valid copy via pg_backup_stop(). This solves torn reads without needing to
>> write pg_control via a temp file, which may affect performance on a standby.
> 
> We're talking about a 8kB file which has a size of 512B
> (PG_CONTROL_MAX_SAFE_SIZE) to avoid such issues.  So I'm not sure to
> see your point here?

Even at 512B it is possible to see tears in pg_control and they happen 
in the build farm right now. In fact, this thread [1] trying to fix the 
problem was what got me thinking about alternate solutions to preventing 
tears in pg_control. Thomas' proposed fixes have not been committed to 
my knowledge so the problem remains, but would be fixed by this commit.

> There is a large comment block in do_pg_backup_stop() around
> backup_stopped_in_recovery.  Perhaps it should be improved based on
> this patch.

I added a sentence to this comment block in v2.

> The main concern that I have over this patch is: who is actually going
> to use this extension of the SQL stop function?

Primarily existing backup software, I would imagine. The idea is that it 
would give them feature parity with pg_basebackup, rather than the new 
protections being exclusive to pg_basebackup.

> Perhaps existing
> backup solutions are good enough risk vs reward is not worth it?  

I'm not sure I see the risk here. Saving out pg_control is optional so 
no changes to current software is required. Of course they miss the 
benefit of the protection against tears and missing backup_label, but 
that is a choice.

Also, no matter what current backup solutions do, they cannot prevent a 
user from removing backup_label after restore. This patch prevents 
invalid recovery when that happens.

> The
> label_file and the tablespace map are text, this is a series of bytes
> that has no visibility for the end-user unless checked on the
> client-side.  This adds a new step where backups would need to copy
> the control file to the data folder.

Again, optional, but if I was able to manage these saves using the psql 
interface in the TAP tests then I'd say it would be pretty easy for 
anyone with a normal connection to Postgres. Also, we require users to 
treat tabelspace_map and backup_label as binary so not too big a change 
here.

[1] 
https://www.postgresql.org/message-id/CA%2BhUKG%2Bjig%2BQdBETj_ab%2B%2BVWSoJjbwT3sLNCnk0wFsY_6tRqoA%...

Attachments:

pgcontrol-from-backupstop-v2.patch (text/plain, 23.8K) download From f9ebd108a9c5dc55fa485dc8c60eb11d2c16715c Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 2 Oct 2024 08:59:12 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 18 +++++- doc/src/sgml/func.sgml | 10 ++- src/backend/access/transam/xlog.c | 23 ++++++- src/backend/access/transam/xlogfuncs.c | 17 ++++-- src/backend/access/transam/xlogrecovery.c | 10 ++- src/backend/backup/basebackup.c | 34 +++++------ src/backend/catalog/system_functions.sql | 2 +- src/backend/utils/misc/pg_controldata.c | 7 ++- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlogbackup.h | 8 +++ src/include/catalog/pg_control.h | 4 ++ src/include/catalog/pg_proc.dat | 10 +-- src/test/recovery/t/002_archiving.pl | 20 ++++++ src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 16 files changed, 192 insertions(+), 42 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index e4e4c56cf1..2fcf181121 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1021,9 +1021,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1095,7 +1098,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index d6acdd3059..a2622014ae 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27899,6 +27899,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> @@ -28576,8 +28581,9 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 The result of the function is a single record. The <parameter>lsn</parameter> column holds the backup's ending write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be + column returns the contents of the backup label file, the third column + returns the contents of the tablespace map file, and the fourth column + returns the contents of pg_control. These must be stored as part of the backup and are required as part of the restore process. </para> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9102c8d772..d28f155700 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9110,7 +9110,9 @@ get_backup_status(void) * wait for WAL segments to be archived. * * "state" is filled with the information necessary to restore from this - * backup with its stop LSN (stoppoint), its timeline ID (stoptli), etc. + * backup with its stop LSN (stoppoint), its timeline ID (stoptli), etc. This + * includes a copy of pg_control with safeguards against it being used without + * backup_label. * * It is the responsibility of the caller of this function to verify the * permissions of the calling user! @@ -9127,11 +9129,30 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) int seconds_before_warning; int waits = 0; bool reported_waiting = false; + ControlFileData *controlFileCopy = (ControlFileData *)state->controlFile; Assert(state != NULL); backup_stopped_in_recovery = RecoveryInProgress(); + /* + * Create a copy of control data and update it to require a backup label + * for recovery. Also recalculate the CRC. + */ + memset( + state->controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupLabelRequired = true; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 3e3d2bb618..fc2401d859 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -115,9 +115,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains represents a consistent copy of pg_control that also has a safeguard + * against being used without backup_label. It is the caller's responsibility + * to write the backup_label, pg_control, and tablespace_map files in the data + * folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -125,12 +127,13 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -152,9 +155,15 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 320b14add1..193b7e2045 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -704,7 +704,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to succeed"))); /* * If tablespace_map file is present without backup_label file, there @@ -977,6 +984,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 14e5ba72e9..d0d4e74cb0 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -325,7 +325,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; @@ -348,16 +347,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, /* Then the bulk of the files... */ sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid, ib); - - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); } else { @@ -374,7 +363,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, * include the xlog files below and stop afterwards. This is safe * since the main data directory is always sent *last*. */ - if (opt->includewal && ti->path == NULL) + if (ti->path == NULL) { Assert(lnext(state.tablespaces, lc) == NULL); } @@ -394,6 +383,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, basebackup_progress_wait_wal_archive(&state); do_pg_backup_stop(backup_state, !opt->nowait); + /* Send pg_control */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_FILE_SIZE, &manifest); + endptr = backup_state->stoppoint; endtli = backup_state->stoptli; @@ -632,16 +626,16 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, StatusFilePath(pathbuf, fname, ".done"); sendFileWithContent(sink, pathbuf, "", -1, &manifest); } + } - /* Properly terminate the tar file. */ - StaticAssertStmt(2 * TAR_BLOCK_SIZE <= BLCKSZ, - "BLCKSZ too small for 2 tar blocks"); - memset(sink->bbs_buffer, 0, 2 * TAR_BLOCK_SIZE); - bbsink_archive_contents(sink, 2 * TAR_BLOCK_SIZE); + /* Properly terminate the tar file. */ + StaticAssertStmt(2 * TAR_BLOCK_SIZE <= BLCKSZ, + "BLCKSZ too small for 2 tar blocks"); + memset(sink->bbs_buffer, 0, 2 * TAR_BLOCK_SIZE); + bbsink_archive_contents(sink, 2 * TAR_BLOCK_SIZE); - /* OK, that's the end of the archive. */ - bbsink_end_archive(sink); - } + /* OK, that's the end of the archive. */ + bbsink_end_archive(sink); AddWALInfoToBackupManifest(&manifest, state.startptr, state.starttli, endptr, endtli); diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index b0d0de051e..758f90b2c3 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 98c932dc7b..9eaf3f8b9f 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93a05d80ca..8cc29904c6 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -283,6 +283,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index e9dcb5a6d8..7056752cff 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -875,6 +875,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 960916a1e8..79a44b70ee 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -722,6 +722,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h index c30d4a5991..93a0e5c5cc 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" /* Structure to hold backup state. */ @@ -35,6 +36,13 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. backupLabelRequired + * has been set so backup_label will be required for recovery to start. + */ + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; } BackupState; extern char *build_backup_content(BackupState *state, diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index e80ff8e414..b471a9b02e 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -164,12 +164,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to succeed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 77f54a79e6..ea93241eed 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6569,8 +6569,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', @@ -12119,9 +12119,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index bc447330e1..462f1dcf0d 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 61d23187e0..bd3a99960f 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2024-10-03 04:45  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2024-10-03 04:45 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: pgsql-hackers

On Wed, Oct 02, 2024 at 09:03:27AM +0000, David Steele wrote:
> On 10/2/24 10:11, Michael Paquier wrote:
>> Hmm, okay.  There is also a slight impact for BASE_BACKUP, requiring
>> basebackup_progress_wait_wal_archive() and do_pg_backup_stop() to be
>> called earlier when sending the main data directory which is the last
>> one in the list of tablespaces.  As far as I can see, this does not
>> change the logic because do_pg_backup_stop() does not touch the
>> control file, but it seems to me that we'd rather keep these two
>> calls as they are now, and send the control file once we are out of
>> the for loop that processes all the tablespaces.  That seems slightly
>> cleaner to me, still I agree that both are the same things.
> 
> Sending pg_control later results in even more code churn because of how tar
> files are terminated. I've updated it that way in v2 so you can see what I
> mean. I don't have a strong preference, though, so if you prefer the
> implementation in v2 then that's fine with me.

It does not make much of a difference, indeed.

>> Anyway, finishing do_pg_backup_stop() and then sending the control
>> file is just a consequence of the implementation choice driven by the
>> output required for the SQL function so as this is stored in the
>> backup state to get it down to pg_backup_stop() in xlogfuncs.c, so we
>> could also take one step back and forget about the SQL function,
>> setting only the new flag when sending a BASE_BACKUP, or just not use
>> the backupState to store this data as the exercise involves forcing
>> one boolean and recalculate a CRC32.
> 
> I can definitely see us making other updates to pg_control so I would rather
> keep this logic centralized, even though it is not too complicated at this
> point. Still, even 8 lines of code (as it is now) seems better not to
> duplicate.

I was wondering if the field update should be hidden behind a macro
that uses an offsetof() on ControlFileData, with the name of the field
and a pointer to the value to update to.  If you include the CRC32
calculation in that, that makes for less chunks of code when updating
one field of the control file.

The full CRC calculation could also be hidden inside a macro, as there
are a couple of places where we do the same things, like pg_rewind.c,
etc.

>> We're talking about a 8kB file which has a size of 512B
>> (PG_CONTROL_MAX_SAFE_SIZE) to avoid such issues.  So I'm not sure to
>> see your point here?
> 
> Even at 512B it is possible to see tears in pg_control and they happen in
> the build farm right now. In fact, this thread [1] trying to fix the problem
> was what got me thinking about alternate solutions to preventing tears in
> pg_control. Thomas' proposed fixes have not been committed to my knowledge
> so the problem remains, but would be fixed by this commit.

Ah, right.  That rings a bell.  Thomas has done some work with
c558e6fd92ff and 63a582222c6b.  And we're still not taking the
ControlFileLock while copying it over..  It looks like we should do it
separately, and backpatch.  That's not something for this thread to
worry about.

>> Perhaps existing
>> backup solutions are good enough risk vs reward is not worth it?
> 
> I'm not sure I see the risk here. Saving out pg_control is optional so no
> changes to current software is required. Of course they miss the benefit of
> the protection against tears and missing backup_label, but that is a choice.
>
> Again, optional, but if I was able to manage these saves using the psql
> interface in the TAP tests then I'd say it would be pretty easy for anyone
> with a normal connection to Postgres. Also, we require users to treat
> tabelspace_map and backup_label as binary so not too big a change here.

Maintenance cost for a limited user impact overall.  With incremental
backups being a thing in v18 only available through the replication
protocol, the SQL functions have less advantages these days.  My point
would be to see this thread as a two-step process:
1) Update the flag in the control file when sending it across in
replication stream.
2) Do the SQL function thing with the bytea for the control file, if
necessary.

1) is something that has more value than 2), IMO, because there is no
need for a manual step when a backup is taken by the replication
protocol.  Well, custom backup solutions that rely on the replication
protocol to copy the data would need to make sure that they have a
backup_label, but that's something they should do anyway and what this
patch wants to protect users from.  The SQL part is optional IMO.  It
can be done, but it has less impact overall and makes backups more
complicated by requiring the manual copy of the control file. 
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2024-10-03 09:11  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2024-10-03 09:11 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 10/3/24 07:45, Michael Paquier wrote:
> On Wed, Oct 02, 2024 at 09:03:27AM +0000, David Steele wrote:
>> On 10/2/24 10:11, Michael Paquier wrote:
> 
>> I can definitely see us making other updates to pg_control so I would rather
>> keep this logic centralized, even though it is not too complicated at this
>> point. Still, even 8 lines of code (as it is now) seems better not to
>> duplicate.
> 
> I was wondering if the field update should be hidden behind a macro
> that uses an offsetof() on ControlFileData, with the name of the field
> and a pointer to the value to update to.  If you include the CRC32
> calculation in that, that makes for less chunks of code when updating
> one field of the control file.
> 
> The full CRC calculation could also be hidden inside a macro, as there
> are a couple of places where we do the same things, like pg_rewind.c,
> etc.

This seems to be a different case than pg_rewind, especially since we 
need a ControlFileLock. I think that is a bit much to do in a macro, so 
I split the functionality out into a function instead. This simplifies 
the logic in basebackup.c but has little impact elsewhere.

>>> We're talking about a 8kB file which has a size of 512B
>>> (PG_CONTROL_MAX_SAFE_SIZE) to avoid such issues.  So I'm not sure to
>>> see your point here?
>>
>> Even at 512B it is possible to see tears in pg_control and they happen in
>> the build farm right now. In fact, this thread [1] trying to fix the problem
>> was what got me thinking about alternate solutions to preventing tears in
>> pg_control. Thomas' proposed fixes have not been committed to my knowledge
>> so the problem remains, but would be fixed by this commit.
> 
> Ah, right.  That rings a bell.  Thomas has done some work with
> c558e6fd92ff and 63a582222c6b.  And we're still not taking the
> ControlFileLock while copying it over..  It looks like we should do it
> separately, and backpatch.  That's not something for this thread to
> worry about.

I'd be happy to adapt patch 01 to be back-patched (without the new flag) 
if we decide it is a good idea. Just locking and making a copy of 
pg_control is easy enough, but if we accept the backup_control_file() 
function for new versions then we could keep that for the back patch to 
reduce churn between versions.

>>> Perhaps existing
>>> backup solutions are good enough risk vs reward is not worth it?
>>
>> I'm not sure I see the risk here. Saving out pg_control is optional so no
>> changes to current software is required. Of course they miss the benefit of
>> the protection against tears and missing backup_label, but that is a choice.
>>
>> Again, optional, but if I was able to manage these saves using the psql
>> interface in the TAP tests then I'd say it would be pretty easy for anyone
>> with a normal connection to Postgres. Also, we require users to treat
>> tabelspace_map and backup_label as binary so not too big a change here.
> 
> Maintenance cost for a limited user impact overall.  With incremental
> backups being a thing in v18 only available through the replication
> protocol, the SQL functions have less advantages these days.  My point
> would be to see this thread as a two-step process:
> 1) Update the flag in the control file when sending it across in
> replication stream.
> 2) Do the SQL function thing with the bytea for the control file, if
> necessary.

OK, I have split the patch into two parts along these lines.

> 1) is something that has more value than 2), IMO, because there is no
> need for a manual step when a backup is taken by the replication
> protocol.  Well, custom backup solutions that rely on the replication
> protocol to copy the data would need to make sure that they have a
> backup_label, but that's something they should do anyway and what this
> patch wants to protect users from.  The SQL part is optional IMO.  It
> can be done, but it has less impact overall and makes backups more
> complicated by requiring the manual copy of the control file.

I don't think having incremental backup in pg_basebackup means alternate 
backup solutions are going away or that we should deprecate the SQL 
interface. If nothing else, third-party solutions need a way to get an 
untorn copy of pg_control and in general I think the new flag will be 
universally useful.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v3-01-basebackup.patch (text/plain, 11.4K) download From 77df2c779b57dde90f81d5244b55e4d0d644137d Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Thu, 3 Oct 2024 08:41:45 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 7b4fbb5047..786269b920 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27899,6 +27899,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9102c8d772..79065b55ef 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9412,6 +9412,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 320b14add1..193b7e2045 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -704,7 +704,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to succeed"))); /* * If tablespace_map file is present without backup_label file, there @@ -977,6 +984,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 14e5ba72e9..ce7fd76e48 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -325,9 +326,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -350,14 +351,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 98c932dc7b..9eaf3f8b9f 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93a05d80ca..8cc29904c6 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -283,6 +283,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index e9dcb5a6d8..7056752cff 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -875,6 +875,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 960916a1e8..79a44b70ee 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -722,6 +722,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 34ad46c067..9d5d8ed43c 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -295,6 +295,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index e80ff8e414..b471a9b02e 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -164,12 +164,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to succeed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 77f54a79e6..7cc5421ea1 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12119,9 +12119,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index bc447330e1..462f1dcf0d 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v3-02-sql.patch (text/plain, 11.3K) download From dba5a2a57e5472682a9be6d4badf1e8127768610 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Thu, 3 Oct 2024 08:41:45 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- doc/src/sgml/func.sgml | 5 +- src/backend/access/transam/xlogfuncs.c | 17 ++++-- src/backend/catalog/system_functions.sql | 2 +- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 6 files changed, 98 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index e4e4c56cf1..2fcf181121 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1021,9 +1021,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1095,7 +1098,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 786269b920..09204db7cd 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28581,8 +28581,9 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 The result of the function is a single record. The <parameter>lsn</parameter> column holds the backup's ending write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be + column returns the contents of the backup label file, the third column + returns the contents of the tablespace map file, and the fourth column + returns the contents of pg_control. These must be stored as part of the backup and are required as part of the restore process. </para> diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 3e3d2bb618..a7d7c1bcbe 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -115,9 +115,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -125,12 +127,13 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -152,9 +155,15 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + backup_control_file((uint8_t *)VARDATA(pg_control_bytea)); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index b0d0de051e..758f90b2c3 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7cc5421ea1..ea93241eed 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6569,8 +6569,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 61d23187e0..bd3a99960f 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2024-11-20 22:44  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2024-11-20 22:44 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 10/3/24 05:11, David Steele wrote:
> On 10/3/24 07:45, Michael Paquier wrote:
> 
>> 1) is something that has more value than 2), IMO, because there is no
>> need for a manual step when a backup is taken by the replication
>> protocol.  Well, custom backup solutions that rely on the replication
>> protocol to copy the data would need to make sure that they have a
>> backup_label, but that's something they should do anyway and what this
>> patch wants to protect users from.  The SQL part is optional IMO.  It
>> can be done, but it has less impact overall and makes backups more
>> complicated by requiring the manual copy of the control file.
> 
> I don't think having incremental backup in pg_basebackup means alternate 
> backup solutions are going away or that we should deprecate the SQL 
> interface. If nothing else, third-party solutions need a way to get an 
> untorn copy of pg_control and in general I think the new flag will be 
> universally useful.

I updated this patch to fix an issue with -fsanitize=alignment. I'm not 
entirely happy with copying twice but not sure of another way to do it. 
As far as I can see VARDATA() will not return aligned data on 64-bit 
architectures.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v4-02-sql.patch (text/plain, 11.4K) download From 659a1d9b6b1528e139741dc69146848ca15a07b9 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 20 Nov 2024 22:33:36 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- doc/src/sgml/func.sgml | 5 +- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/backend/catalog/system_functions.sql | 2 +- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 6 files changed, 101 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index e4e4c56cf14..2fcf1811213 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1021,9 +1021,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1095,7 +1098,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index fa37a0469ae..b3d8c73295b 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28600,8 +28600,9 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 The result of the function is a single record. The <parameter>lsn</parameter> column holds the backup's ending write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be + column returns the contents of the backup label file, the third column + returns the contents of the tablespace map file, and the fourth column + returns the contents of pg_control. These must be stored as part of the backup and are required as part of the restore process. </para> diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index b0c6d7c6875..7547bb6e28e 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -113,9 +113,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -123,12 +125,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -150,9 +154,17 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index c51dfca802f..6a29b9b3cf3 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 8d9d584735e..6f65cc7c777 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6589,8 +6589,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 61d23187e0f..bd3a99960f5 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
pgcontrol-flag-v4-01-basebackup.patch (text/plain, 11.4K) download From 005365fd1f87a03b4e4de8485be0ed6af9127acc Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 20 Nov 2024 22:33:36 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 72f223a0414..fa37a0469ae 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27918,6 +27918,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 6f58412bcab..2a2c80a6f74 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9413,6 +9413,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 05c738d6614..f641cb1a8ad 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -703,7 +703,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to succeed"))); /* * If tablespace_map file is present without backup_label file, there @@ -976,6 +983,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index e2ed9081d1c..b673bc10bc3 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -326,9 +327,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -351,14 +352,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 98c932dc7bd..9eaf3f8b9f6 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93a05d80ca7..8cc29904c6b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -283,6 +283,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index e9dcb5a6d89..7056752cff0 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -875,6 +875,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index c4fe4e37040..b9faa51661e 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -725,6 +725,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 34ad46c067b..9d5d8ed43c2 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -295,6 +295,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index e80ff8e4140..b471a9b02e7 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -164,12 +164,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to succeed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index cbbe8acd382..8d9d584735e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12139,9 +12139,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index bc447330e15..462f1dcf0d6 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2025-01-24 18:43  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2025-01-24 18:43 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 11/20/24 17:44, David Steele wrote:
> On 10/3/24 05:11, David Steele wrote:
>> On 10/3/24 07:45, Michael Paquier wrote:
>>
>>> 1) is something that has more value than 2), IMO, because there is no
>>> need for a manual step when a backup is taken by the replication
>>> protocol.  Well, custom backup solutions that rely on the replication
>>> protocol to copy the data would need to make sure that they have a
>>> backup_label, but that's something they should do anyway and what this
>>> patch wants to protect users from.  The SQL part is optional IMO.  It
>>> can be done, but it has less impact overall and makes backups more
>>> complicated by requiring the manual copy of the control file.
>>
>> I don't think having incremental backup in pg_basebackup means 
>> alternate backup solutions are going away or that we should deprecate 
>> the SQL interface. If nothing else, third-party solutions need a way 
>> to get an untorn copy of pg_control and in general I think the new 
>> flag will be universally useful.
> 
> I updated this patch to fix an issue with -fsanitize=alignment. I'm not 
> entirely happy with copying twice but not sure of another way to do it. 
> As far as I can see VARDATA() will not return aligned data on 64-bit 
> architectures.

Rebased and improved a comment and an error.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v5-02-sql.patch (text/plain, 11.4K) download From b229466927f4c17c2c058dbaca92b8ec0429bd80 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 24 Jan 2025 18:38:09 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- doc/src/sgml/func.sgml | 5 +- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/backend/catalog/system_functions.sql | 2 +- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 6 files changed, 101 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index e4e4c56cf14..2fcf1811213 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1021,9 +1021,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1095,7 +1098,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 86c946905aa..61907e088d7 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28671,8 +28671,9 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 The result of the function is a single record. The <parameter>lsn</parameter> column holds the backup's ending write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be + column returns the contents of the backup label file, the third column + returns the contents of the tablespace map file, and the fourth column + returns the contents of pg_control. These must be stored as part of the backup and are required as part of the restore process. </para> diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 8c3090165f0..e805f231c69 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -113,9 +113,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -123,12 +125,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -150,9 +154,17 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 591157b1d1b..fa635a698a3 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 0416569b823..dbd8a89d9d9 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6602,8 +6602,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 5749a1df533..a08a9a9d9df 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
pgcontrol-flag-v5-01-basebackup.patch (text/plain, 11.4K) download From 8d9c268780c2128bb96d1bbdb2762aeca5b2393c Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 24 Jan 2025 18:38:08 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 5678e7621a5..86c946905aa 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27989,6 +27989,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index bf3dbda901d..9285a4a5c72 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9413,6 +9413,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index cf2b007806f..72ced660f88 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -703,7 +703,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed"))); /* * If tablespace_map file is present without backup_label file, there @@ -976,6 +983,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 3f8a3c55725..a457f5714d1 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -326,9 +327,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -351,14 +352,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 9dfba499c13..983be8496a2 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93a05d80ca7..8cc29904c6b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -283,6 +283,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index ed73607a46f..9fb3bbebc42 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -875,6 +875,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index cae81cd6cb1..26fcdc1cd71 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -725,6 +725,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 4411c1468ac..59c80b65165 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -295,6 +295,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 3797f25b306..b7d864e103a 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -164,12 +164,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 18560755d26..0416569b823 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12172,9 +12172,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index 3acdb9ff1eb..153c658c123 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2025-08-06 22:30  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2025-08-06 22:30 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 1/24/25 13:43, David Steele wrote:
> 
> Rebased and improved a comment and an error.
Rebased to fix breakage caused by the split of func.sgml in 4e23c9e.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v6-01-basebackup.patch (text/plain, 11.5K) download From c44525f64f6a2eed4f2636db70e255237244d5b4 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 6 Aug 2025 22:25:41 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index b507bfaf64b..f185c80cec2 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3588,6 +3588,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9a4de1616bc..04922836cbc 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9579,6 +9579,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index f23ec8969c2..9efbafce335 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -709,7 +709,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed"))); /* * If tablespace_map file is present without backup_label file, there @@ -984,6 +991,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index bb7d90aa5d9..b2b6b259b69 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -326,9 +327,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -351,14 +352,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 6d036e3bf32..d0ebd8e98a7 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 10de058ce91..16046b3f094 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -294,6 +294,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index e876f35f38e..be23e50d1c1 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -899,6 +899,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = 0; ControlFile.backupEndPoint = 0; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 0c68dd4235e..d365132eff6 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -727,6 +727,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index d12798be3d8..4d0137096d4 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -295,6 +295,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 63e834a6ce4..84df7ee6f50 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -164,12 +164,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 118d6da1ace..7d05cf89eec 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12326,9 +12326,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index 3acdb9ff1eb..153c658c123 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v6-02-sql.patch (text/plain, 10.4K) download From 8e5f815ccfa313d7c59649e9baa473b63c21862b Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 6 Aug 2025 22:25:42 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/backend/catalog/system_functions.sql | 2 +- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 5 files changed, 98 insertions(+), 13 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 5f7489afbd1..7ea85172d51 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1027,9 +1027,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1101,7 +1104,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 8c3090165f0..e805f231c69 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -113,9 +113,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -123,12 +125,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -150,9 +154,17 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 566f308e443..f8833393833 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -390,7 +390,7 @@ CREATE OR REPLACE FUNCTION CREATE OR REPLACE FUNCTION pg_backup_stop ( wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + OUT labelfile text, OUT spcmapfile text, OUT controlfile bytea) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7d05cf89eec..f15bc5260ee 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6700,8 +6700,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 5749a1df533..a08a9a9d9df 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-02-20 03:10  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-02-20 03:10 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 8/7/25 05:30, David Steele wrote:
> On 1/24/25 13:43, David Steele wrote:
>>
>> Rebased and improved a comment and an error.
> Rebased to fix breakage caused by the split of func.sgml in 4e23c9e.


Rebased to implement simplification added by "Simplify creation of 
built-in functions with default arguments" (759b03b2).

Regards,
-David





^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-02-20 05:47  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-02-20 05:47 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 2/20/26 10:10, David Steele wrote:
> On 8/7/25 05:30, David Steele wrote:
>> On 1/24/25 13:43, David Steele wrote:
>>>
>>> Rebased and improved a comment and an error.
>> Rebased to fix breakage caused by the split of func.sgml in 4e23c9e.
> 
> 
> Rebased to implement simplification added by "Simplify creation of 
> built-in functions with default arguments" (759b03b2).

With the patches this time!

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v7-01-basebackup.patch (text/plain, 11.5K) download From f4043472a7acb4c74566ac17026d0e4b6245aa58 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 20 Feb 2026 02:59:52 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 294f45e82a3..2e78d28221c 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3646,6 +3646,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..babb76b4710 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9586,6 +9586,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c0c2744d45b..18a0db611e6 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -710,7 +710,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed"))); /* * If tablespace_map file is present without backup_label file, there @@ -995,6 +1002,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 2d74c648335..f54c8793090 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -331,9 +332,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -356,14 +357,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index c6d9cbb1577..c2c19eb77df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index a4060309ae0..5919dd58fed 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -301,6 +301,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 85dc43d4cdb..5eddebe27f2 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -918,6 +918,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d0aafd7e7a6..d5b49b2a26b 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -736,6 +736,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..84ec5aabdfd 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -311,6 +311,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 7503db1af51..26618162a7b 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -166,12 +166,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index dac40992cbc..3c368433615 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12437,9 +12437,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index aa40f58e6d6..9963d13473e 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v7-02-sql.patch (text/plain, 9.7K) download From e654c44aac81c2aa5b70b03a62067b0caedcae1c Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 20 Feb 2026 02:59:52 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 4 files changed, 97 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..67974b7b1b6 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1027,9 +1027,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1101,7 +1104,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 2efe4105efb..e7e7c0b18e5 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -113,9 +113,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -123,12 +125,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -150,9 +154,17 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 3c368433615..c89967613a5 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6741,8 +6741,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', proargdefaults => '{true}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index df4ae029fe6..34c6f4461af 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-06 01:27  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-03-06 01:27 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers

On 2/20/26 12:47, David Steele wrote:
> On 2/20/26 10:10, David Steele wrote:
>> On 8/7/25 05:30, David Steele wrote:
>>> On 1/24/25 13:43, David Steele wrote:
>>>>
>>>> Rebased and improved a comment and an error.
>>> Rebased to fix breakage caused by the split of func.sgml in 4e23c9e.
>>
>>
>> Rebased to implement simplification added by "Simplify creation of 
>> built-in functions with default arguments" (759b03b2).

Rebased on "Simplify creation of built-in functions with non-default 
ACLs." (f95d73ed).

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v8-01-basebackup.patch (text/plain, 11.5K) download From a38d5e68b648e1d03832c3b917a60a2a5ed4c61b Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 6 Mar 2026 01:10:14 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 +++++ src/backend/access/transam/xlog.c | 23 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 10 +++++++++- src/backend/backup/basebackup.c | 15 ++++++--------- src/backend/utils/misc/pg_controldata.c | 7 +++++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 ++++ src/include/catalog/pg_proc.dat | 6 +++--- src/test/recovery/t/002_archiving.pl | 20 ++++++++++++++++++++ 12 files changed, 80 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 294f45e82a3..2e78d28221c 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3646,6 +3646,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 354ac645bdc..66f0d7e091d 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9563,6 +9563,29 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile + sizeof(ControlFileData), 0, + PG_CONTROL_FILE_SIZE - sizeof(ControlFileData)); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index fbddd7e522c..71759ae67b3 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -646,7 +646,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed"))); /* * If tablespace_map file is present without backup_label file, there @@ -931,6 +938,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 2d74c648335..f54c8793090 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -331,9 +332,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -356,14 +357,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index c6d9cbb1577..c2c19eb77df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index a4060309ae0..5919dd58fed 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -301,6 +301,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index ab766c34d4b..768a4ed2f18 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -918,6 +918,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 9d745d4b25b..509b9e80a21 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -736,6 +736,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..84ec5aabdfd 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -311,6 +311,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 7503db1af51..26618162a7b 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -166,12 +166,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4950bff2804..ece9b4f08c9 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12494,9 +12494,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index aa40f58e6d6..9963d13473e 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v8-02-sql.patch (text/plain, 9.7K) download From 1de5c13947884e33f744177d7f21147826a180e1 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Fri, 6 Mar 2026 01:10:14 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 4 files changed, 97 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..67974b7b1b6 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1027,9 +1027,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1101,7 +1104,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 2efe4105efb..e7e7c0b18e5 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -113,9 +113,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -123,12 +125,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -150,9 +154,17 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Build the contents of backup_label */ backup_label = build_backup_content(backup_state, false); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index ece9b4f08c9..9a85bc987df 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6757,8 +6757,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', proargdefaults => '{true}', prosrc => 'pg_backup_stop', proacl => '{POSTGRES=X}' }, diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index df4ae029fe6..34c6f4461af 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-17 05:16  Haibo Yan <tristan.yim@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 2 replies; 64+ messages in thread

From: Haibo Yan @ 2026-03-17 05:16 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hi David
I have not read the code yet, so this may already be answered there, but I had a question about the proposal itself. This patch protects against a missing backup_label, but what about a wrong one? If a user restores a backup_label file from a different backup, the existence check alone would not detect that. Do we need some consistency check between the returned pg_control copy and the backup_label contents, or is the intended scope here limited to the “missing file” case only?
Regards
Haibo

> On Mar 5, 2026, at 5:27 PM, David Steele <david@pgbackrest.org> wrote:
> 
> On 2/20/26 12:47, David Steele wrote:
>> On 2/20/26 10:10, David Steele wrote:
>>> On 8/7/25 05:30, David Steele wrote:
>>>> On 1/24/25 13:43, David Steele wrote:
>>>>> 
>>>>> Rebased and improved a comment and an error.
>>>> Rebased to fix breakage caused by the split of func.sgml in 4e23c9e.
>>> 
>>> 
>>> Rebased to implement simplification added by "Simplify creation of built-in functions with default arguments" (759b03b2).
> 
> Rebased on "Simplify creation of built-in functions with non-default ACLs." (f95d73ed).
> 
> Regards,
> -David<pgcontrol-flag-v8-01-basebackup.patch><pgcontrol-flag-v8-02-sql.patch>

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-17 05:51  Michael Paquier <michael@paquier.xyz>
  parent: Haibo Yan <tristan.yim@gmail.com>
  1 sibling, 0 replies; 64+ messages in thread

From: Michael Paquier @ 2026-03-17 05:51 UTC (permalink / raw)
  To: Haibo Yan <tristan.yim@gmail.com>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers

On Mon, Mar 16, 2026 at 10:16:58PM -0700, Haibo Yan wrote:
> I have not read the code yet, so this may already be answered there,
> but I had a question about the proposal itself. This patch protects
> against a missing backup_label, but what about a wrong one? If a
> user restores a backup_label file from a different backup, the
> existence check alone would not detect that. Do we need some
> consistency check between the returned pg_control copy and the
> backup_label contents, or is the intended scope here limited to the
> “missing file” case only? 

Please note that we use bottom-posting on the lists, as of:
https://en.wikipedia.org/wiki/Posting_style#Bottom-posting
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-17 07:05  David Steele <david@pgbackrest.org>
  parent: Haibo Yan <tristan.yim@gmail.com>
  1 sibling, 1 reply; 64+ messages in thread

From: David Steele @ 2026-03-17 07:05 UTC (permalink / raw)
  To: Haibo Yan <tristan.yim@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

On 3/17/26 12:16, Haibo Yan wrote:

> I have not read the code yet, so this may already be answered there, but 
> I had a question about the proposal itself. This patch protects against 
> a missing backup_label, but what about a wrong one? If a user restores a 
> backup_label file from a different backup, the existence check alone 
> would not detect that. Do we need some consistency check between the 
> returned pg_control copy and the backup_label contents, or is the 
> intended scope here limited to the “missing file” case only?

Thank you for having a look!

The goal here is only to check for a missing backup_label. The general 
problem is that PostgreSQL suggests that removing backup_label might be 
a good idea so the user does it:

If you are not restoring from a backup, try removing the file 
\"%s/backup_label\"

The user *could* copy a backup_label from another backup and there are 
ways we could detect that but I feel that should be material for a 
separate patch.

Regards,
-David







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-17 18:50  Haibo Yan <tristan.yim@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Haibo Yan @ 2026-03-17 18:50 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers

Hi David,

Thank you for the clarification. I have now read the code, and overall it looks good to me. I only had one very small comment.

You currently have:
```
memset(controlFile + sizeof(ControlFileData), 0,
       PG_CONTROL_FILE_SIZE - sizeof(ControlFileData));
memcpy(controlFile, ControlFile, sizeof(ControlFileData));
``` 
This is correct, since only the trailing bytes need to be zeroed before the copy.

I was just wondering whether the following might be slightly clearer:
```
memset(controlFile, 0, PG_CONTROL_FILE_SIZE);
memcpy(controlFile, ControlFile, sizeof(ControlFileData));
```

I do not think this is a real issue, though.

Thanks
Haibo

> On Mar 17, 2026, at 12:05 AM, David Steele <david@pgbackrest.org> wrote:
> 
> On 3/17/26 12:16, Haibo Yan wrote:
> 
>> I have not read the code yet, so this may already be answered there, but I had a question about the proposal itself. This patch protects against a missing backup_label, but what about a wrong one? If a user restores a backup_label file from a different backup, the existence check alone would not detect that. Do we need some consistency check between the returned pg_control copy and the backup_label contents, or is the intended scope here limited to the “missing file” case only?
> 
> Thank you for having a look!
> 
> The goal here is only to check for a missing backup_label. The general problem is that PostgreSQL suggests that removing backup_label might be a good idea so the user does it:
> 
> If you are not restoring from a backup, try removing the file \"%s/backup_label\"
> 
> The user *could* copy a backup_label from another backup and there are ways we could detect that but I feel that should be material for a separate patch.
> 
> Regards,
> -David
> 
> 

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 01:43  Michael Paquier <michael@paquier.xyz>
  parent: Haibo Yan <tristan.yim@gmail.com>
  0 siblings, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-03-18 01:43 UTC (permalink / raw)
  To: Haibo Yan <tristan.yim@gmail.com>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On Tue, Mar 17, 2026 at 11:50:29AM -0700, Haibo Yan wrote:
> Thank you for the clarification. I have now read the code, and
> overall it looks good to me. I only had one very small comment.

(Bottom-posting note from above, please be careful.)

> I was just wondering whether the following might be slightly clearer:
> ```
> memset(controlFile, 0, PG_CONTROL_FILE_SIZE);
> memcpy(controlFile, ControlFile, sizeof(ControlFileData));
> ```
> 
> I do not think this is a real issue, though.

         {
             ControlFile->backupStartPoint = checkPoint.redo;
             ControlFile->backupEndRequired = backupEndRequired;
+            ControlFile->backupLabelRequired = false;

It sounds like it is going to be important to document the reason why
the flag is reset here (aka we don't need the backup_label file
anymore).

+backup_control_file(uint8_t *controlFile)
+{
+    ControlFileData *controlData = ((ControlFileData *)controlFile);
+
+    memset(controlFile + sizeof(ControlFileData), 0,
+           PG_CONTROL_FILE_SIZE - sizeof(ControlFileData));
+
+    LWLockAcquire(ControlFileLock, LW_SHARED);
+    memcpy(controlFile, ControlFile, sizeof(ControlFileData));
+    LWLockRelease(ControlFileLock);
+
+    controlData->backupLabelRequired = true;
+
+    INIT_CRC32C(controlData->crc);
+    COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc));
+    FIN_CRC32C(controlData->crc);

I was wondering if we should have an assertion at least to cross-check
that the contents we store in shared memory never go out-of-sync with
the on-disk contents, in the shape of a USE_ASSERT_CHECKING block that
calls get_controlfile() and memcmp()'s the contents between shmem and
the on-disk file, while the LWLock is taken.  We ship the control file
last on purpose, one reason being backups taken from standbys, so that
may be sensible to do.

Another property of the new control file flag that is implied in the
implementation but not documented is that we should never check for
backupLabelRequired when a backup_label is gone.  Actually, the flag
is reset in InitWalRecovery() in the initial steps of recovery, and
the backup_label file is removed much later in StartupXLOG() just
*after* UpdateControlFile() to minimize the window where the contents
of the control file and the backup_label file is removed are
out-of-sync.  This window means that if we crash between the
completion of UpdateControlFile() and the durable_rename() we could
have a flag reset with a backup_label still around.  On restart,
recovery would fail, requiring a manual modification of the control
file, at least.  It sounds to me that this implementation detail
should be documented clearly.

Finally, here is a general opinion.  I like this patch, and it is
basically risk-free for base backups taken with the replication
protocol as we update the control file with the new flag set
on-the-fly. 

Now, I am worried about backups that use pg_stop_backup().  Changing
backup APIs has always been a very sensitive area, and this is going
to require operators to update backup tools so as the control file
received as a result of pg_stop_backup() is copied, at the cost of
getting a failure if they don't do so.  I will *not* proceed with this
change without a clear approval from some more committers or senior
hackers that they like this change (approach previously suggested by
Andres, actually, for what I can see).  I am adding in CC a few
committers who have commented on this set of proposals and who have
touched the recovery code in the last few years, for awareness.
The timing is what it is, and we are at the end of a release cycle.
Let's see if we can reach a consensus of some kind.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 04:05  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-03-18 04:05 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; +Cc: pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 3/18/26 08:43, Michael Paquier wrote:
> On Tue, Mar 17, 2026 at 11:50:29AM -0700, Haibo Yan wrote:
>> Thank you for the clarification. I have now read the code, and
>> overall it looks good to me. I only had one very small comment.
> 
> (Bottom-posting note from above, please be careful.)
> 
>> I was just wondering whether the following might be slightly clearer:
>> ```
>> memset(controlFile, 0, PG_CONTROL_FILE_SIZE);
>> memcpy(controlFile, ControlFile, sizeof(ControlFileData));
>> ```

Yeah, perhaps I am being too clever here. The reason why I like this 
pattern:

memset(controlFile + sizeof(ControlFileData), 0,
        PG_CONTROL_FILE_SIZE - sizeof(ControlFileData));

...is that valgrind will complain if the ControlFileData part is not 
completely initialized later on.

But your version is what is generally used in the code so I'm fine with 
doing it that way.

>           {
>               ControlFile->backupStartPoint = checkPoint.redo;
>               ControlFile->backupEndRequired = backupEndRequired;
> +            ControlFile->backupLabelRequired = false;
> 
> It sounds like it is going to be important to document the reason why
> the flag is reset here (aka we don't need the backup_label file
> anymore).

Good point -- I'll add that in the next revision.

> +backup_control_file(uint8_t *controlFile)
> +{
> +    ControlFileData *controlData = ((ControlFileData *)controlFile);
> +
> +    memset(controlFile + sizeof(ControlFileData), 0,
> +           PG_CONTROL_FILE_SIZE - sizeof(ControlFileData));
> +
> +    LWLockAcquire(ControlFileLock, LW_SHARED);
> +    memcpy(controlFile, ControlFile, sizeof(ControlFileData));
> +    LWLockRelease(ControlFileLock);
> +
> +    controlData->backupLabelRequired = true;
> +
> +    INIT_CRC32C(controlData->crc);
> +    COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc));
> +    FIN_CRC32C(controlData->crc);
> 
> I was wondering if we should have an assertion at least to cross-check
> that the contents we store in shared memory never go out-of-sync with
> the on-disk contents, in the shape of a USE_ASSERT_CHECKING block that
> calls get_controlfile() and memcmp()'s the contents between shmem and
> the on-disk file, while the LWLock is taken.  We ship the control file
> last on purpose, one reason being backups taken from standbys, so that
> may be sensible to do.

As far as I can see this should always be true -- I audited all the

LWLockAcquire(ControlFileLock, LW_EXCLUSIVE)

sections and the file is always saved once if is updated. Let me see if 
I can add this check without too much pain, e.g. an additional parameter.

> Another property of the new control file flag that is implied in the
> implementation but not documented is that we should never check for
> backupLabelRequired when a backup_label is gone.  

I'm not sure what you mean here? That's exactly when we do want to check 
as below:

/*
     * No backup_label file has been found if we are here. Error if the
     * control file requires backup_label.
     */
if (ControlFile->backupLabelRequired)
     ereport(FATAL,
             (errmsg("could not find backup_label required for recovery"),
                 errhint("backup_label must be present for recovery to 
proceed")));


 > Actually, the flag> is reset in InitWalRecovery() in the initial 
steps of recovery, and
> the backup_label file is removed much later in StartupXLOG() just
> *after* UpdateControlFile() to minimize the window where the contents
> of the control file and the backup_label file is removed are
> out-of-sync.  This window means that if we crash between the
> completion of UpdateControlFile() and the durable_rename() we could
> have a flag reset with a backup_label still around.  On restart,
> recovery would fail, requiring a manual modification of the control
> file, at least.  It sounds to me that this implementation detail
> should be documented clearly.

I'll test it but I don't think this is the case. If backup_label is 
present then we'll just update pg_control again as we do now.

When backup_label is present the value of backupLabelRequired does not 
matter so it just follows the prior logic.

> Finally, here is a general opinion.  I like this patch, and it is
> basically risk-free for base backups taken with the replication
> protocol as we update the control file with the new flag set
> on-the-fly.

Glad to hear it!

> Now, I am worried about backups that use pg_stop_backup().  Changing
> backup APIs has always been a very sensitive area, and this is going
> to require operators to update backup tools so as the control file
> received as a result of pg_stop_backup() is copied, at the cost of
> getting a failure if they don't do so.  

Using the pg_control copy from pg_backup_stop() is entirely optional and 
nothing is broken if vendors decide not to use it. This can be 
demonstrated by applying the 01 patch without 02. In that case the tests 
in 042_low_level_backup continue to run. You can also apply 01 and 02 
and revert the test changes in 042_low_level_backup and that works, too.

Vendors can use the new feature if they want the protection of 
backupLabelRequired and a guaranteed non-torn copy of pg_control but if 
they do nothing then nothing breaks.

> I will *not* proceed with this
> change without a clear approval from some more committers or senior
> hackers that they like this change (approach previously suggested by
> Andres, actually, for what I can see).  

You are correct and this was an omission on my part. If this gets to 
commit we'll definitely want to mention that this flag was Andres' 
suggestion.

> I am adding in CC a few
> committers who have commented on this set of proposals and who have
> touched the recovery code in the last few years, for awareness.
> The timing is what it is, and we are at the end of a release cycle.
> Let's see if we can reach a consensus of some kind.

Perfect. I'm looking forward to their input.

I'll hold off on a new patch version until we get some feedback since I 
don't think any of the requested changes are critical to the functionality.

Regards,
-David





^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 04:53  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-03-18 04:53 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On Wed, Mar 18, 2026 at 04:05:24AM +0000, David Steele wrote:
> On 3/18/26 08:43, Michael Paquier wrote:
>> I was wondering if we should have an assertion at least to cross-check
>> that the contents we store in shared memory never go out-of-sync with
>> the on-disk contents, in the shape of a USE_ASSERT_CHECKING block that
>> calls get_controlfile() and memcmp()'s the contents between shmem and
>> the on-disk file, while the LWLock is taken.  We ship the control file
>> last on purpose, one reason being backups taken from standbys, so that
>> may be sensible to do.
> 
> As far as I can see this should always be true -- I audited all the
> 
> LWLockAcquire(ControlFileLock, LW_EXCLUSIVE)
> 
> sections and the file is always saved once if is updated. Let me see if I
> can add this check without too much pain, e.g. an additional parameter.

This matches with my reads of the code.  The attached check, that can
be applied on top of your patches, passes under check-world.

>> Another property of the new control file flag that is implied in the
>> implementation but not documented is that we should never check for
>> backupLabelRequired when a backup_label is gone.
> 
> I'm not sure what you mean here? That's exactly when we do want to check as
> below:

Sorry for the confusion, I meant that "we should never check for
backupLabelRequired when we have a backup_label".

> Using the pg_control copy from pg_backup_stop() is entirely optional and
> nothing is broken if vendors decide not to use it. This can be demonstrated
> by applying the 01 patch without 02. In that case the tests in
> 042_low_level_backup continue to run. You can also apply 01 and 02 and
> revert the test changes in 042_low_level_backup and that works, too.

FWIW, after a second look I am actually wondering if 0002 is safe at
all.  The contents of the control file are fetched after we are done
with do_pg_backup_stop(), and there could be a bunch of activity that
happens between the end of do_pg_backup_stop() and the
backup_control_file() call, where the control file could be updated
more and interfere with the recovery startup for some of its fields?
GUC parameter updates that may touch the control file are one thing
popping into mind.
--
Michael

Attachments:

0001-Add-consistency-check-for-shmem-and-on-disk-control-.patch (text/plain, 1005B) download From de5b9b00867e39e73dc99a1ee61a814cead1ab7d Mon Sep 17 00:00:00 2001 From: Michael Paquier <michael@paquier.xyz> Date: Wed, 18 Mar 2026 13:44:01 +0900 Subject: [PATCH] Add consistency check for shmem and on-disk control file --- src/backend/access/transam/xlog.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index c8ea1ea49d24..0f8516fcb563 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9606,6 +9606,17 @@ backup_control_file(uint8_t *controlFile) LWLockAcquire(ControlFileLock, LW_SHARED); memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, ControlFile, sizeof(ControlFileData)) == 0); + } +#endif + LWLockRelease(ControlFileLock); controlData->backupLabelRequired = true; -- 2.53.0
signature.asc (application/pgp-signature, 832B) download ^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 07:35  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-03-18 07:35 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 3/18/26 11:53, Michael Paquier wrote:
> On Wed, Mar 18, 2026 at 04:05:24AM +0000, David Steele wrote:
>> On 3/18/26 08:43, Michael Paquier wrote:
>>> I was wondering if we should have an assertion at least to cross-check
>>> that the contents we store in shared memory never go out-of-sync with
>>> the on-disk contents, in the shape of a USE_ASSERT_CHECKING block that
>>> calls get_controlfile() and memcmp()'s the contents between shmem and
>>> the on-disk file, while the LWLock is taken.  We ship the control file
>>> last on purpose, one reason being backups taken from standbys, so that
>>> may be sensible to do.
>>
>> As far as I can see this should always be true -- I audited all the
>>
>> LWLockAcquire(ControlFileLock, LW_EXCLUSIVE)
>>
>> sections and the file is always saved once if is updated. Let me see if I
>> can add this check without too much pain, e.g. an additional parameter.
> 
> This matches with my reads of the code.  The attached check, that can
> be applied on top of your patches, passes under check-world.

Looks good to me. I have integrated it into the attached patches. I just 
changed it to look at our copy instead of the global ControlData since 
that is the one we are going to save.

>> Using the pg_control copy from pg_backup_stop() is entirely optional and
>> nothing is broken if vendors decide not to use it. This can be demonstrated
>> by applying the 01 patch without 02. In that case the tests in
>> 042_low_level_backup continue to run. You can also apply 01 and 02 and
>> revert the test changes in 042_low_level_backup and that works, too.
> 
> FWIW, after a second look I am actually wondering if 0002 is safe at
> all.  The contents of the control file are fetched after we are done
> with do_pg_backup_stop(), and there could be a bunch of activity that
> happens between the end of do_pg_backup_stop() and the
> backup_control_file() call, where the control file could be updated
> more and interfere with the recovery startup for some of its fields?
> GUC parameter updates that may touch the control file are one thing
> popping into mind.

You are correct -- the copy of pg_control needs to happen before 
do_pg_backup_stop(). An older version of this patch saved pg_control in 
backup_state which made the prior location safe. However, I missed 
moving this code when I moved pg_control out of backup_state. Code 
review to the rescue.

I believe I have addressed all current review comments in the attached 
patches. I tested Postgres crashing right after pg_control is updated 
but before backup_label is renamed. It worked as expected on restart. I 
also manually removed backup_label before restarting and that worked as 
well. I wrote a comment documenting all that.

One thing I could do is note in the documentation that it is not 
strictly necessary to get pg_control from pg_backup_stop(). Right now it 
sounds like copying from disk is no longer an option -- but it is if you 
are willing to accept the possibility of pg_control being torn. But I'd 
hope most well-maintained backup software is taking care of that by now.

The problem with caveats in the docs is it can lead to confusion and 
getting pg_control from pg_backup_stop() is just a better idea in general.

Thoughts?

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v9-01-basebackup.patch (text/plain, 12.4K) download From 8694c31dee28606448edfa5b0375fad8a7d619bd Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 18 Mar 2026 07:11:13 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 ++++ src/backend/access/transam/xlog.c | 36 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 19 +++++++++++- src/backend/backup/basebackup.c | 15 ++++------ src/backend/utils/misc/pg_controldata.c | 7 +++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 +++ src/include/catalog/pg_proc.dat | 6 ++-- src/test/recovery/t/002_archiving.pl | 20 +++++++++++++ 12 files changed, 102 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 5b5f1f3c5df..b1b49ce7b07 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3661,6 +3661,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index f5c9a34374d..87ee245e3f4 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9592,6 +9592,42 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + /* + * Verify that the contents of pg_control are the same in memory as on disk + */ + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, controlFile, sizeof(ControlFileData)) == 0); + } +#endif + + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 6d2c4a86b96..bd702e895d7 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -647,7 +647,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + (errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed"))); /* * If tablespace_map file is present without backup_label file, there @@ -927,11 +934,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index ab1fbae8001..c4f7827f866 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -332,9 +333,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -357,14 +358,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index c6d9cbb1577..c2c19eb77df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index a4060309ae0..5919dd58fed 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -301,6 +301,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index ab766c34d4b..768a4ed2f18 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -918,6 +918,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 9d745d4b25b..509b9e80a21 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -736,6 +736,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index dcc12eb8cbe..469739aa23f 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -313,6 +313,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 77a661e818b..d9841f58108 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -166,12 +166,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index fc8d82665b8..999ea2ac801 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12504,9 +12504,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index aa40f58e6d6..9963d13473e 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v9-02-sql.patch (text/plain, 9.8K) download From 5e19f24aa151e7502799e26c5d9ac7d574b4750f Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Wed, 18 Mar 2026 07:11:13 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 4 files changed, 97 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..67974b7b1b6 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1027,9 +1027,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1101,7 +1104,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 4e35311b2f3..fab0c047621 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -143,9 +143,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -153,12 +155,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -174,6 +178,13 @@ pg_backup_stop(PG_FUNCTION_ARGS) Assert(backup_state != NULL); Assert(tablespace_map != NULL); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); @@ -183,6 +194,7 @@ pg_backup_stop(PG_FUNCTION_ARGS) values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 999ea2ac801..ce717e8828c 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6767,8 +6767,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', proargdefaults => '{true}', prosrc => 'pg_backup_stop', proacl => '{POSTGRES=X}' }, diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index df4ae029fe6..34c6f4461af 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 08:26  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-03-18 08:26 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On Wed, Mar 18, 2026 at 07:35:47AM +0000, David Steele wrote:
> You are correct -- the copy of pg_control needs to happen before
> do_pg_backup_stop(). An older version of this patch saved pg_control in
> backup_state which made the prior location safe. However, I missed moving
> this code when I moved pg_control out of backup_state. Code review to the
> rescue.

Right.  I am wondering also if the final result would not be better
without 0002, actually, focusing only on the "simpler" base backup
case through the replication protocol, and you are making a good case
in mentioning it as not absolutely mandatory for base backups that are
taken through the SQL functions.  One could always tweak the flag
manually in the control file based on the contents taken from the data
folder.  That's more hairy than writing the entire file, for sure,
still possible.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 12:26  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 3 replies; 64+ messages in thread

From: David Steele @ 2026-03-18 12:26 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 3/18/26 15:26, Michael Paquier wrote:
> On Wed, Mar 18, 2026 at 07:35:47AM +0000, David Steele wrote:
>> You are correct -- the copy of pg_control needs to happen before
>> do_pg_backup_stop(). An older version of this patch saved pg_control in
>> backup_state which made the prior location safe. However, I missed moving
>> this code when I moved pg_control out of backup_state. Code review to the
>> rescue.
> 
> Right.  I am wondering also if the final result would not be better
> without 0002, actually, focusing only on the "simpler" base backup
> case through the replication protocol, and you are making a good case
> in mentioning it as not absolutely mandatory for base backups that are
> taken through the SQL functions.  One could always tweak the flag
> manually in the control file based on the contents taken from the data
> folder.  That's more hairy than writing the entire file, for sure,
> still possible.

Getting even 01 into PG19 would be a great outcome. This would solve the 
problem of torn pg_control and deleted backup labels for any backups 
made with pg_basebackup and that's going to cover a *lot* of cases.

Established third-party backup solutions that are not based on 
pg_basebackup are generally able to manipulate pg_control so that's not 
as much of a concern, perhaps. It does raise the barrier of entry for 
new backup software if they need to learn to read and validate 
pg_control to avoid a torn copy and set the flag. Patch 02 solves that 
problem in a general way so I still think it adds value for the 
ecosystem -- but we could always discuss that in the PG20 cycle.

Whatever gets committed for PG19 I'll write a followup patch to describe 
the hazards of reading pg_control and generally how to get a good copy. 
However, this will be complicated enough that the best answer will 
likely be to use pg_basebackup or some other reputable backup software. 
I don't love this -- I feel like the low-level interface should be 
usable with such hazards.

Regards,
-David





^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-18 21:00  Corey Huinker <corey.huinker@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  2 siblings, 1 reply; 64+ messages in thread

From: Corey Huinker @ 2026-03-18 21:00 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

>
> Whatever gets committed for PG19 I'll write a followup patch to describe
> the hazards of reading pg_control and generally how to get a good copy.
> However, this will be complicated enough that the best answer will
> likely be to use pg_basebackup or some other reputable backup software.
> I don't love this -- I feel like the low-level interface should be
> usable with such hazards.


Surya Poondla and I had decided on this patchset as a pair-reviewing
exercise. However, events have overtaken us, and several other people have
chimed in expressing the same concerns that we had observed but hadn't yet
completed our review. All of the main concerns that we had found up to this
point have been addressed in the lastest patchset, except for the trivial
observation that the ereport() uses the old style and doesn't need the set
of parens around (errmsg(), errhint()). Patches apply clean, tests pass,
test coverage seems sufficient, we're happy with the wording of the
documentation, in short there really isn't a whole lot for us to add to the
review, and for that reason we're removing our names from the list of
reviewers in the commitfest app.

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-19 02:55  David Steele <david@pgbackrest.org>
  parent: Corey Huinker <corey.huinker@gmail.com>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-03-19 02:55 UTC (permalink / raw)
  To: Corey Huinker <corey.huinker@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

Hi Corey,

On 3/19/26 04:00, Corey Huinker wrote:
>     Whatever gets committed for PG19 I'll write a followup patch to
>     describe
>     the hazards of reading pg_control and generally how to get a good copy.
>     However, this will be complicated enough that the best answer will
>     likely be to use pg_basebackup or some other reputable backup software.
>     I don't love this -- I feel like the low-level interface should be
>     usable with such hazards.
> 
> Surya Poondla and I had decided on this patchset as a pair-reviewing 
> exercise. However, events have overtaken us, and several other people 
> have chimed in expressing the same concerns that we had observed but 
> hadn't yet completed our review. 

Thank you both for having a look!

 > All of the main concerns that we had > found up to this point have 
been addressed in the lastest patchset,
> except for the trivial observation that the ereport() uses the old style 
> and doesn't need the set of parens around (errmsg(), errhint()). 

Grep shows there are lots of messages with the new style but many more 
in the old style. Presumably they are only being updated as they are 
modified. Do you happen to know the commit or message thread where this 
policy was started? I've been searching but it is such a generic search 
term.

I've updated the message style in the new patches.

> Patches 
> apply clean, tests pass, test coverage seems sufficient, we're happy 
> with the wording of the documentation, in short there really isn't a 
> whole lot for us to add to the review, and for that reason we're 
> removing our names from the list of reviewers in the commitfest app.

It seems to me you've still done a review. Confirming what the other 
reviewers found is good info to have.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-flag-v10-01-basebackup.patch (text/plain, 12.4K) download From 79f696a8936b05849bf38a5d393d93c642a2450b Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Thu, 19 Mar 2026 02:50:51 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 ++++ src/backend/access/transam/xlog.c | 36 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 19 +++++++++++- src/backend/backup/basebackup.c | 15 ++++------ src/backend/utils/misc/pg_controldata.c | 7 +++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 +++ src/include/catalog/pg_proc.dat | 6 ++-- src/test/recovery/t/002_archiving.pl | 20 +++++++++++++ 12 files changed, 102 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 5b5f1f3c5df..b1b49ce7b07 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3661,6 +3661,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index f5c9a34374d..87ee245e3f4 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -9592,6 +9592,42 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + */ +void +backup_control_file(uint8_t *controlFile) +{ + ControlFileData *controlData = ((ControlFileData *)controlFile); + + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFile, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + /* + * Verify that the contents of pg_control are the same in memory as on disk + */ + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, controlFile, sizeof(ControlFileData)) == 0); + } +#endif + + LWLockRelease(ControlFileLock); + + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlFile, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 6d2c4a86b96..e2810d68a0a 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -647,7 +647,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("backup_label must be present for recovery to proceed")); /* * If tablespace_map file is present without backup_label file, there @@ -927,11 +934,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index ab1fbae8001..c4f7827f866 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -332,9 +333,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -357,14 +358,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index c6d9cbb1577..c2c19eb77df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index a4060309ae0..5919dd58fed 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -301,6 +301,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index ab766c34d4b..768a4ed2f18 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -918,6 +918,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 9d745d4b25b..509b9e80a21 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -736,6 +736,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index dcc12eb8cbe..469739aa23f 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -313,6 +313,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8_t *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 77a661e818b..d9841f58108 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -166,12 +166,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index fc8d82665b8..999ea2ac801 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12504,9 +12504,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index aa40f58e6d6..9963d13473e 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
pgcontrol-flag-v10-02-sql.patch (text/plain, 9.8K) download From b7ed57808d81a2d1fcc09487fb7a566a515f2351 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Thu, 19 Mar 2026 02:50:52 +0000 Subject: Return pg_control from pg_backup_stop(). Harden recovery by returning a copy of pg_control from pg_backup_stop() that has a flag set to prevent recovery if the backup_label file is missing. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If backup_label is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. These changes have no impact on current backup software and they are free to use the pg_control available from pg_stop_backup() or continue to use pg_control from PGDATA. Of course they will miss the benefits of getting a consistent copy of pg_control and the backup_label checking, but will be no worse off than before. Catalog version bump is required. --- doc/src/sgml/backup.sgml | 18 +++++- src/backend/access/transam/xlogfuncs.c | 20 ++++-- src/include/catalog/pg_proc.dat | 4 +- src/test/recovery/t/042_low_level_backup.pl | 67 ++++++++++++++++++++- 4 files changed, 97 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..67974b7b1b6 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1027,9 +1027,12 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); values. The second of these fields should be written to a file named <filename>backup_label</filename> in the root directory of the backup. The third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <filename>tablespace_map</filename> unless the field is empty. The fourth + field should be written into a file named + <filename>global/pg_control</filename> (overwriting the existing file when + present). These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> @@ -1101,7 +1104,16 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You should exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>controlfile</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This version of pg_control contains + safeguards against recovery without backup_label present and is guaranteed + not to be torn. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 4e35311b2f3..fab0c047621 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -143,9 +143,11 @@ pg_backup_start(PG_FUNCTION_ARGS) * * The backup_label contains the user-supplied label string (typically this * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * time, starting WAL location for the dump and so on. The pg_control file + * contains a consistent copy of pg_control that also has a safeguard against + * being used without backup_label. It is the caller's responsibility to write + * the backup_label, pg_control, and tablespace_map files in the data folder + * that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -153,12 +155,14 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 4 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); char *backup_label; + bytea *pg_control_bytea; + uint8_t pg_control[PG_CONTROL_FILE_SIZE]; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -174,6 +178,13 @@ pg_backup_stop(PG_FUNCTION_ARGS) Assert(backup_state != NULL); Assert(tablespace_map != NULL); + /* Build the contents of pg_control */ + backup_control_file(pg_control); + + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), pg_control, PG_CONTROL_FILE_SIZE); + /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); @@ -183,6 +194,7 @@ pg_backup_stop(PG_FUNCTION_ARGS) values[0] = LSNGetDatum(backup_state->stoppoint); values[1] = CStringGetTextDatum(backup_label); values[2] = CStringGetTextDatum(tablespace_map->data); + values[3] = PointerGetDatum(pg_control_bytea); /* Deallocate backup-related variables */ pfree(backup_label); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 999ea2ac801..ce717e8828c 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6767,8 +6767,8 @@ { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,pg_lsn,text,text,bytea}', proargmodes => '{i,o,o,o,o}', + proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile,controlfile}', proargdefaults => '{true}', prosrc => 'pg_backup_stop', proacl => '{POSTGRES=X}' }, diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index df4ae029fe6..34c6f4461af 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -13,6 +13,42 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Decode hex to binary +sub decode_hex +{ + my ($encoded) = @_; + my $decoded; + + $encoded =~ s/^\s+|\s+$//g; + + for (my $idx = 0; $idx < length($encoded); $idx += 2) + { + $decoded .= pack('C', hex(substr($encoded, $idx, 2))); + } + + return $decoded; +} + +# Get backup_label/pg_control from pg_stop_backup() +sub stop_backup_result +{ + my ($psql) = @_; + + my $encoded = $psql->query_safe( + "select encode(labelfile::bytea, 'hex') || ',' || " . + " encode(controlfile, 'hex')" . + " from pg_backup_stop()"); + + my @result; + + foreach my $column (split(',', $encoded)) + { + push(@result, decode_hex($column)); + } + + return @result; +} + # Start primary node with archiving. my $node_primary = PostgreSQL::Test::Cluster->new('primary'); $node_primary->init(has_archiving => 1, allows_streaming => 1); @@ -80,8 +116,7 @@ my $stop_segment_name = $node_primary->safe_psql('postgres', 'SELECT pg_walfile_name(pg_current_wal_lsn())'); # Stop backup and get backup_label, the last segment is archived. -my $backup_label = - $psql->query_safe("select labelfile from pg_backup_stop()"); +(my $backup_label, my $pg_control) = stop_backup_result($psql); $psql->quit; @@ -118,10 +153,36 @@ ok( $node_replica->log_contains( $node_replica->teardown_node; $node_replica->clean_node; +# Save only pg_control into the backup to demonstrate that pg_control returned +# from pg_stop_backup() will only perform recovery when backup_label is present. +open(my $fh, ">", "$backup_dir/global/pg_control") + or die "could not open pg_control"; +binmode($fh); +syswrite($fh, $pg_control); +close($fh); + +$node_replica = PostgreSQL::Test::Cluster->new('replica_fail2'); +$node_replica->init_from_backup($node_primary, $backup_name, + has_restoring => 1); + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_replica->data_dir, '-l', + $node_replica->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_replica->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +$node_replica->teardown_node; +$node_replica->clean_node; + # Save backup_label into the backup directory and recover using the primary's # archive. This time recovery will succeed and the canary table will be # present. -open my $fh, ">>", "$backup_dir/backup_label" +open $fh, ">>", "$backup_dir/backup_label" or die "could not open backup_label"; # Binary mode is required for Windows, as the backup_label parsing is not # able to cope with CRLFs. -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-03-19 04:20  Corey Huinker <corey.huinker@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 0 replies; 64+ messages in thread

From: Corey Huinker @ 2026-03-19 04:20 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On Wed, Mar 18, 2026 at 10:56 PM David Steele <david@pgbackrest.org> wrote:

> Hi Corey,
>
> On 3/19/26 04:00, Corey Huinker wrote:
> >     Whatever gets committed for PG19 I'll write a followup patch to
> >     describe
> >     the hazards of reading pg_control and generally how to get a good
> copy.
> >     However, this will be complicated enough that the best answer will
> >     likely be to use pg_basebackup or some other reputable backup
> software.
> >     I don't love this -- I feel like the low-level interface should be
> >     usable with such hazards.
> >
> > Surya Poondla and I had decided on this patchset as a pair-reviewing
> > exercise. However, events have overtaken us, and several other people
> > have chimed in expressing the same concerns that we had observed but
> > hadn't yet completed our review.
>
> Thank you both for having a look!
>
>  > All of the main concerns that we had > found up to this point have
> been addressed in the lastest patchset,
> > except for the trivial observation that the ereport() uses the old style
> > and doesn't need the set of parens around (errmsg(), errhint()).
>
> Grep shows there are lots of messages with the new style but many more
> in the old style. Presumably they are only being updated as they are
> modified.


That's always been my assumption. Not worth the churn.


> Do you happen to know the commit or message thread where this
> policy was started? I've been searching but it is such a generic search
> term.
>

I limited my git log -p to elog.h, and it seems it started with
e3a87b4991cc back in 2020. The only reason I knew about it was that I used
to do backports from v13 to unsupported versions, and the new style would
cause the build to fail on an otherwise clean cherry pick.


> It seems to me you've still done a review. Confirming what the other
> reviewers found is good info to have.
>

Of a sort, yes, but our review doesn't touch the "is this a good idea"
question, which has been by far the thing most in need of reviewing across
the long discussion threads.

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-04-13 14:55  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  2 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-04-13 14:55 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 3/18/26 19:26, David Steele wrote:
> On 3/18/26 15:26, Michael Paquier wrote:
>> On Wed, Mar 18, 2026 at 07:35:47AM +0000, David Steele wrote:
>>> You are correct -- the copy of pg_control needs to happen before
>>> do_pg_backup_stop(). An older version of this patch saved pg_control in
>>> backup_state which made the prior location safe. However, I missed 
>>> moving
>>> this code when I moved pg_control out of backup_state. Code review to 
>>> the
>>> rescue.
>>
>> Right.  I am wondering also if the final result would not be better
>> without 0002, actually, focusing only on the "simpler" base backup
>> case through the replication protocol, and you are making a good case
>> in mentioning it as not absolutely mandatory for base backups that are
>> taken through the SQL functions.  One could always tweak the flag
>> manually in the control file based on the contents taken from the data
>> folder.  That's more hairy than writing the entire file, for sure,
>> still possible.
> 
> Getting even 01 into PG19 would be a great outcome. This would solve the 
> problem of torn pg_control and deleted backup labels for any backups 
> made with pg_basebackup and that's going to cover a *lot* of cases.
> 
> Established third-party backup solutions that are not based on 
> pg_basebackup are generally able to manipulate pg_control so that's not 
> as much of a concern, perhaps. It does raise the barrier of entry for 
> new backup software if they need to learn to read and validate 
> pg_control to avoid a torn copy and set the flag. Patch 02 solves that 
> problem in a general way so I still think it adds value for the 
> ecosystem -- but we could always discuss that in the PG20 cycle.
> 
> Whatever gets committed for PG19 I'll write a followup patch to describe 
> the hazards of reading pg_control and generally how to get a good copy. 
> However, this will be complicated enough that the best answer will 
> likely be to use pg_basebackup or some other reputable backup software. 
> I don't love this -- I feel like the low-level interface should be 
> usable with such hazards.

I have withdrawn this patch. If anybody wants to pick it up in the 
future I'll be happy to rebase it but I think two years is long enough 
to maintain a patch that is not getting traction.

We are left with the issue that pg_basebackup backups may contain a torn 
copy of pg_control. At the least this should be documented.

It would also be a good idea to document that utilizing the low-level 
backup interface requires validating the checksum in pg_control to avoid 
a torn copy. This is non-trivial but certainly doable.

Regards,
-David





^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-06-30 05:15  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-06-30 05:15 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 4/13/26 21:55, David Steele wrote:
> 
> I have withdrawn this patch. If anybody wants to pick it up in the 
> future I'll be happy to rebase it but I think two years is long enough 
> to maintain a patch that is not getting traction.
I would like to revive this but focus on the first patch for now and 
drop the second patch from consideration.

The patch now implements only the new flag for pgcontrol and wires this 
logic into basebackup. It has the additional benefit of guaranteeing 
that the base backup contains a non-torn version of pgcontrol.

I know everyone was really busy in March but now that things are a bit 
calmer I'd like to revisit.

Heikki, Robert, Andres, Fujii -- any objections or comments? I believe 
Michael is on board with the feature (this part, at least) but he very 
sensibly would like to have some consensus.

Thanks,
-David

Attachments:

pgcontrol-basebackup-flag-v11.patch (text/plain, 12.8K) download From 6859828e85c7d54ab71b4dc3e01c45a47b89c469 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgmasters.net> Date: Tue, 30 Jun 2026 04:42:47 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup (or any software comfortable with modifying pg_control). Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 +++ src/backend/access/transam/xlog.c | 44 +++++++++++++++++++++++ src/backend/access/transam/xlogrecovery.c | 19 +++++++++- src/backend/backup/basebackup.c | 15 ++++---- src/backend/utils/misc/pg_controldata.c | 7 ++-- src/bin/pg_controldata/pg_controldata.c | 2 ++ src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 +++ src/include/catalog/pg_proc.dat | 6 ++-- src/test/recovery/t/002_archiving.pl | 20 +++++++++++ 12 files changed, 110 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 69ef3857cfa..58d7e9adb19 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3649,6 +3649,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index a81912b7441..7bfd193c59b 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10074,6 +10074,50 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * aligned for ControlFileData (e.g. it may point into the payload of a bytea). + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + /* + * Verify that the contents of pg_control are the same in memory as on disk + */ + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, &controlData, sizeof(ControlFileData)) == 0); + + pfree(dataDisk); + } +#endif + + LWLockRelease(ControlFileLock); + + controlData.backupLabelRequired = true; + + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c0ae4d3f63f..5668d13d788 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -648,7 +648,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -928,11 +935,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 9c79dadaacc..4730ec2ac65 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -332,9 +333,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -357,14 +358,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index c6d9cbb1577..c2c19eb77df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -162,8 +162,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -195,6 +195,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index fe5fc5ec133..cfe19f4090a 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -303,6 +303,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index f8d25afed9d..848ed8bcdb0 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -918,6 +918,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 9d745d4b25b..509b9e80a21 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -736,6 +736,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 4dd98624204..182b2fe8f7c 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -326,6 +326,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 80b3a730e03..2ad1254adf1 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1a985becde3..dd20fe1402a 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12354,9 +12354,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/t/002_archiving.pl b/src/test/recovery/t/002_archiving.pl index aa40f58e6d6..9963d13473e 100644 --- a/src/test/recovery/t/002_archiving.pl +++ b/src/test/recovery/t/002_archiving.pl @@ -41,6 +41,26 @@ $node_standby->append_conf( archive_cleanup_command = 'echo archive_cleanup_done > $archive_cleanup_command_file' recovery_end_command = 'echo recovery_ended_done > $recovery_end_command_file' )); + +# Rename backup_label to verify that recovery will not start without it +rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT "could not move $data_dir/backup_label"; + +my $res = run_log( + [ + 'pg_ctl', '-D', $node_standby->data_dir, '-l', + $node_standby->logfile, 'start' + ]); +ok(!$res, 'invalid recovery startup fails'); + +my $logfile = slurp_file($node_standby->logfile()); +ok($logfile =~ qr/could not find backup_label required for recovery/, + 'could not find backup_label required for recovery'); + +# Restore backup_label so recovery proceeds normally +rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT "could not move $data_dir/backup_label"; + $node_standby->start; # Create some content on primary -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-07-14 10:27  solai v <solai.cdac@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 2 replies; 64+ messages in thread

From: solai v @ 2026-07-14 10:27 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

Hi all,

On Wed, Jul 8, 2026 at 2:49 PM David Steele <david@pgbackrest.org> wrote:
>
> On 4/13/26 21:55, David Steele wrote:
> >
> > I have withdrawn this patch. If anybody wants to pick it up in the
> > future I'll be happy to rebase it but I think two years is long enough
> > to maintain a patch that is not getting traction.
> I would like to revive this but focus on the first patch for now and
> drop the second patch from consideration.
>
> The patch now implements only the new flag for pgcontrol and wires this
> logic into basebackup. It has the additional benefit of guaranteeing
> that the base backup contains a non-torn version of pgcontrol.
>
> I know everyone was really busy in March but now that things are a bit
> calmer I'd like to revisit.
>
> Heikki, Robert, Andres, Fujii -- any objections or comments? I believe
> Michael is on board with the feature (this part, at least) but he very
> sensibly would like to have some consensus.


I reviewed and tested this patch on my PostgreSQL 20devel tree. The
patch applied cleanly and built successfully without any issues. As
part of the testing, I first attempted to reproduce the issue using a
backup created with pg_backup_start()/pg_backup_stop() followed by a
filesystem copy of the data directory. After restoring the backup and
deleting backup_label, the server started successfully.
While in between, I found that backupLabelRequired is set through
backup_control_file(), due to which the above workflow does not
exercise the new code path introduced by this patch. I then repeated
the test using pg_basebackup. After creating a physical backup, I
verified using pg_controldata that the backed-up pg_control contained:

Backup label required: yes.

I continued with the testing like restoring the backup, removing the
backup_label file, and attempting to start the restored cluster. The
PostgreSQL refused to start and reported the errors as expected:

FATAL: could not find backup_label required for recovery
HINT: restore the backup_label file that was created during the backup.

This behavior is consistent with the intended purpose of the patch and
discussions thereby preventing recovery from a pg_basebackup backup
when the required backup_label file is missing. And also I reviewed
the implementation and verified that the backupLabelRequired field is
added to ControlFileData, set while preparing the copied pg_control in
backup_control_file(), checked during recovery, and cleared after
successful recovery. Overall, the patch behaves as intended and looks
good to me.

Regards,
Solai






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-07-20 02:20  David Steele <david@pgbackrest.org>
  parent: solai v <solai.cdac@gmail.com>
  1 sibling, 0 replies; 64+ messages in thread

From: David Steele @ 2026-07-20 02:20 UTC (permalink / raw)
  To: solai v <solai.cdac@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

Hi Solai,

Thank you for having a look at the patch.

On 7/14/26 17:27, solai v wrote:
> On Wed, Jul 8, 2026 at 2:49 PM David Steele <david@pgbackrest.org> wrote:
>>
>> The patch now implements only the new flag for pgcontrol and wires this
>> logic into basebackup. It has the additional benefit of guaranteeing
>> that the base backup contains a non-torn version of pgcontrol.
> 
> I reviewed and tested this patch on my PostgreSQL 20devel tree. The
> patch applied cleanly and built successfully without any issues. As
> part of the testing, I first attempted to reproduce the issue using a
> backup created with pg_backup_start()/pg_backup_stop() followed by a
> filesystem copy of the data directory. After restoring the backup and
> deleting backup_label, the server started successfully.
> While in between, I found that backupLabelRequired is set through
> backup_control_file(), due to which the above workflow does not
> exercise the new code path introduced by this patch. 

I previously had a second patch that would also make 
pg_backup_start()/pg_backup_stop() work with this new feature but 
Michael did not seem in favor of it so I dropped it to try and get the 
first part committed.

> I then repeated
> the test using pg_basebackup. After creating a physical backup, I
> verified using pg_controldata that the backed-up pg_control contained:
> 
> Backup label required: yes.
> 
> I continued with the testing like restoring the backup, removing the
> backup_label file, and attempting to start the restored cluster. The
> PostgreSQL refused to start and reported the errors as expected:
> 
> FATAL: could not find backup_label required for recovery
> HINT: restore the backup_label file that was created during the backup.
> 
> This behavior is consistent with the intended purpose of the patch and
> discussions thereby preventing recovery from a pg_basebackup backup
> when the required backup_label file is missing. And also I reviewed
> the implementation and verified that the backupLabelRequired field is
> added to ControlFileData, set while preparing the copied pg_control in
> backup_control_file(), checked during recovery, and cleared after
> successful recovery. Overall, the patch behaves as intended and looks
> good to me.
Excellent, thank you!

Regards,
-David






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-07-28 10:36  Andy Pogrebnoi <andrew.pogrebnoi@percona.com>
  parent: David Steele <david@pgbackrest.org>
  2 siblings, 1 reply; 64+ messages in thread

From: Andy Pogrebnoi @ 2026-07-28 10:36 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On Wed, Mar 18, 2026 at 2:27 PM David Steele <david@pgbackrest.org> wrote:

> On 3/18/26 15:26, Michael Paquier wrote:
> > On Wed, Mar 18, 2026 at 07:35:47AM +0000, David Steele wrote:
> >> You are correct -- the copy of pg_control needs to happen before
> >> do_pg_backup_stop(). An older version of this patch saved pg_control in
> >> backup_state which made the prior location safe. However, I missed
> moving
> >> this code when I moved pg_control out of backup_state. Code review to
> the
> >> rescue.
> >
> > Right.  I am wondering also if the final result would not be better
> > without 0002, actually, focusing only on the "simpler" base backup
> > case through the replication protocol, and you are making a good case
> > in mentioning it as not absolutely mandatory for base backups that are
> > taken through the SQL functions.  One could always tweak the flag
> > manually in the control file based on the contents taken from the data
> > folder.  That's more hairy than writing the entire file, for sure,
> > still possible.
>
> Getting even 01 into PG19 would be a great outcome. This would solve the
> problem of torn pg_control and deleted backup labels for any backups
> made with pg_basebackup and that's going to cover a *lot* of cases.
>
>
It'd be great to have 01 committed. It seems to me like a solid and low
cost solution for the torn pg_control (and missed backup lables). These are
good improvements for base backups.

I also reviewed the 01 patch but have nothing to add to the previous
reviewers; looks good to me.

---
Cheers,
Andy

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-07-30 22:02  Shaun Thomas <shaun.thomas@pgedge.com>
  parent: Andy Pogrebnoi <andrew.pogrebnoi@percona.com>
  0 siblings, 1 reply; 64+ messages in thread

From: Shaun Thomas @ 2026-07-30 22:02 UTC (permalink / raw)
  To: Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; +Cc: David Steele <david@pgbackrest.org>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

> It'd be great to have 01 committed. It seems to me like a solid and low cost
> solution for the torn pg_control (and missed backup lables). These are good
> improvements for base backups.

I'd like to second this, if possible. It's not exactly a "glamorous" or
critical issue, but this _does_ address a known issue re: torn pg_control if
nothing else. Based on the conversation I saw here, it seems like there's no
impetus so the patch is merely languishing, sans any lingering unaddressed
criticism.

Does it just need a committer to take ownership?

-- 
Shaun Thomas
Principal Software Engineer
shaun.thomas@pgEdge.com






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-08-28 02:34  shihao zhong <zhong950419@gmail.com>
  parent: Shaun Thomas <shaun.thomas@pgedge.com>
  0 siblings, 1 reply; 64+ messages in thread

From: shihao zhong @ 2026-08-28 02:34 UTC (permalink / raw)
  To: Shaun Thomas <shaun.thomas@pgedge.com>; +Cc: Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; David Steele <david@pgbackrest.org>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>

On 6/30/26 05:15, David Steele wrote:
> The patch now implements only the new flag for pgcontrol and wires this
> logic into basebackup.

Hi David,

Thanks for working on this -- it fills a gap that I think Postgres has
needed for a long time.
I reviewed and tested v11 on master and it looks all good.

Attached is v12: your patch plus a TAP test, two one-token fixes and a
commit message tweak, and the same as a delta against v11.

One thing I think needs a decision, which I left alone. xlogrecovery.c
still suggests removing backup_label in two places. Restore a
"pg_basebackup -X none" backup without recovery.signal and you get
"could not locate required checkpoint record", hinting "try removing the
file backup_label". Do that, and you get "could not find backup_label
required for recovery" telling you to put it back.

Blocking startup is right -- that second attempt used to succeed and
silently corrupt the cluster. But the patch exists to stop people from
removing backup_label, so the hint telling them to do so probably should
not survive it. Either suppress those lines when backupLabelRequired is
set, or have the new hint mention recovery.signal, which is usually the
real problem. Say which you prefer and I will write it.

Smaller things:
1. errhint() should start with a capital letter ("Restore the ...").
Fixed in v12, along with "(char *)controlFile" -> "(char *) controlFile".

2. The commit message says "limited to pg_basebackup", but the patch also
sets the flag in pg_rewind. The logic is fine; v12 just says so, and
it probably deserves a line in pg_rewind.sgml.

3. Assert(crc_ok && memcmp(...) == 0) combines two conditions; splitting
them would say which one failed.

4. v12 moves the test out of 002_archiving.pl, which is about
archive_cleanup_command, into 057_backup_label_required.pl. Your
assertions are kept, plus checks that the flag is set in the backup,
cleared after recovery, and the same for a backup from a standby.

5. Docs are untouched beyond the pg_control_recovery() table. At least
pg_basebackup.sgml needs a mention, and something in backup.sgml about
low-level backups not getting this protection now that the SQL part is
dropped. Worth saying that pg_resetwal is the only way to clear the
flag, too.


Thanks,
Shihao

Attachments:

pgcontrol-basebackup-flag-v12.patch (application/octet-stream, 16.9K) download From ab7cfdc1fa59f57db5994a6a7000e74af4be1288 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Thu, 27 Aug 2026 22:13:22 -0400 Subject: [PATCH] Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own, so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Control and catalog version bumps are required. --- doc/src/sgml/func/func-info.sgml | 5 + src/backend/access/transam/xlog.c | 44 +++++++ src/backend/access/transam/xlogrecovery.c | 19 ++- src/backend/backup/basebackup.c | 15 +-- src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + .../recovery/t/057_backup_label_required.pl | 119 ++++++++++++++++++ 13 files changed, 210 insertions(+), 15 deletions(-) create mode 100644 src/test/recovery/t/057_backup_label_required.pl diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index 2f03766b67a..afbbbcab20e 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3659,6 +3659,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index de4c96e135f..a704091e080 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10139,6 +10139,50 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * aligned for ControlFileData (e.g. it may point into the payload of a bytea). + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + /* + * Verify that the contents of pg_control are the same in memory as on disk + */ + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, &controlData, sizeof(ControlFileData)) == 0); + + pfree(dataDisk); + } +#endif + + LWLockRelease(ControlFileLock); + + controlData.backupLabelRequired = true; + + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 6de13b91748..dc7198da248 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -659,7 +659,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +946,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index e3c04ecd810..74ab7781512 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -340,9 +341,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -365,14 +366,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *) controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 9014f0953e9..e328fcc44ff 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6fc87ed114d..200b9777e06 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 1542a56ca4b..7b5e71f4987 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -922,6 +922,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * Force the defaults for max_* settings. The values don't really matter diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 2e86fd158d0..f3f1b15310d 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -738,6 +738,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 130ba929109..340387e7042 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -340,6 +340,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 7b5404460ec..c298e247a21 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 6979c7d1161..ca0b852d958 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12440,9 +12440,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 72113c5ac6e..082dfb17c00 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -65,6 +65,7 @@ tests += { 't/054_unlogged_sequence_promotion.pl', 't/055_cascade_reconnect.pl', 't/056_standby_snapshot_export.pl', + 't/057_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/057_backup_label_required.pl b/src/test/recovery/t/057_backup_label_required.pl new file mode 100644 index 00000000000..79e3a352e92 --- /dev/null +++ b/src/test/recovery/t/057_backup_label_required.pl @@ -0,0 +1,119 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing();
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-15 15:05  David Steele <david@pgbackrest.org>
  parent: shihao zhong <zhong950419@gmail.com>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-09-15 15:05 UTC (permalink / raw)
  To: shihao zhong <zhong950419@gmail.com>; +Cc: Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi Shihao,

On 8/28/26 05:34, shihao zhong wrote:
> 
> Thanks for working on this -- it fills a gap that I think Postgres has
> needed for a long time.
> I reviewed and tested v11 on master and it looks all good.

Thank you for having a look at the patch.

> One thing I think needs a decision, which I left alone. xlogrecovery.c
> still suggests removing backup_label in two places. Restore a
> "pg_basebackup -X none" backup without recovery.signal and you get
> "could not locate required checkpoint record", hinting "try removing the
> file backup_label". Do that, and you get "could not find backup_label
> required for recovery" telling you to put it back.

Agreed, and I have addressed it in v13 by adding a conditional to the 
relevant errors with a alternate message:

ControlFile->backupLabelRequired ?
errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add 
required recovery options.\n"
    "Do not remove \"%s/backup_label\"; it is required to recover this 
backup.",
    DataDir, DataDir, DataDir) :

> Smaller things:
> 1. errhint() should start with a capital letter ("Restore the ...").
> Fixed in v12, along with "(char *)controlFile" -> "(char *) controlFile".

pg_indent would catch this, but always better to have to source as clean 
as possible. Added into v13.

> 2. The commit message says "limited to pg_basebackup", but the patch also
> sets the flag in pg_rewind. The logic is fine; v12 just says so, and
> it probably deserves a line in pg_rewind.sgml.

Changes to the commit message preserved in v13. Docs updated per below.

> 3. Assert(crc_ok && memcmp(...) == 0) combines two conditions; splitting
> them would say which one failed.

I left this as is from v11. I think it's a pretty normal thing to do. If 
you are running with asserts then you are debugging anyway so resolving 
the missed condition is pretty trivial, IMO.

> 4. v12 moves the test out of 002_archiving.pl, which is about
> archive_cleanup_command, into 057_backup_label_required.pl. Your
> assertions are kept, plus checks that the flag is set in the backup,
> cleared after recovery, and the same for a backup from a standby.

Yes, I think a dedicated test module makes sense here, so I kept that in 
v13.

> 5. Docs are untouched beyond the pg_control_recovery() table. At least
> pg_basebackup.sgml needs a mention, and something in backup.sgml about
> low-level backups not getting this protection now that the SQL part is
> dropped. Worth saying that pg_resetwal is the only way to clear the
> flag, too.
Docs have been updated with each of these suggestions.

I also added a few additional tests to flesh out the coverage.

Regards,
-David

Attachments:

pgcontrol-basebackup-flag-v13.patch (text/plain, 26.5K) download From 53c087c4daadc1d88fcad68bc3045ba01b9a6c6e Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Tue, 15 Sep 2026 12:48:33 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own, so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 13 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 10 ++ doc/src/sgml/ref/pg_rewind.sgml | 4 + src/backend/access/transam/xlog.c | 47 +++++ src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/backup/basebackup.c | 15 +- src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/057_backup_label_required.pl | 165 ++++++++++++++++++ 18 files changed, 314 insertions(+), 15 deletions(-) create mode 100644 src/test/recovery/t/057_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..984984b8959 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + Backup software using this API should set the flag itself if it is able to + modify <filename>pg_control</filename>. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index e56c9a22c42..f7ce58479a9 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3644,6 +3644,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index 3117968d125..9bc6210fde2 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,16 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is removed, recovery refuses to start + rather than silently producing an inconsistent cluster. The flag is cleared + automatically once recovery has read <filename>backup_label</filename>; + <xref linkend="app-pgresetwal"/> is the only way to clear it otherwise, and + doing so on a backup that has not been recovered will corrupt it. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index ac3d0c9328f..c4888c183fb 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. + <filename>pg_control</filename> is also flagged to require + <filename>backup_label</filename>, so that removing that file does not + allow the target to start up in an inconsistent state. The flag is + cleared once the target has processed <filename>backup_label</filename>. </para> </step> <step> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9ec0be77ca0..f44651d5eb5 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10589,6 +10589,53 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + * + * The caller's buffer must be at least PG_CONTROL_FILE_SIZE bytes and is + * zero-padded to that size. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * suitably aligned for ControlFileData. + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + +#ifdef USE_ASSERT_CHECKING + /* + * Verify that the contents of pg_control are the same in memory as on disk + */ + { + bool crc_ok; + ControlFileData *dataDisk = get_controlfile(DataDir, &crc_ok); + + Assert(crc_ok && + memcmp(dataDisk, &controlData, sizeof(ControlFileData)) == 0); + + pfree(dataDisk); + } +#endif + + LWLockRelease(ControlFileLock); + + controlData.backupLabelRequired = true; + + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index fff8d57ac61..f61e4c0199b 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index e3c04ecd810..74ab7781512 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -340,9 +341,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -365,14 +366,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *) controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 41afc4c1316..9d36c5b4a84 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -922,6 +922,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 7a590b7e1ea..23f42bc7602 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -340,6 +340,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f46427258e3..19523bc882e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12442,9 +12442,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 72113c5ac6e..082dfb17c00 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -65,6 +65,7 @@ tests += { 't/054_unlogged_sequence_promotion.pl', 't/055_cascade_reconnect.pl', 't/056_standby_snapshot_export.pl', + 't/057_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..0e2cc7ee8ac 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 057_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/057_backup_label_required.pl b/src/test/recovery/t/057_backup_label_required.pl new file mode 100644 index 00000000000..cdf927af596 --- /dev/null +++ b/src/test/recovery/t/057_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal is the only supported way to clear the flag without recovering. +# Use a copy so the original backup is left intact for the restore tests below. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-15 18:13  Andrey Borodin <x4mmm@yandex-team.ru>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Andrey Borodin @ 2026-09-15 18:13 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: shihao zhong <zhong950419@gmail.com>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi David,

On 15 Sep 2026, David Steele wrote:
> I left this as is from v11. I think it's a pretty normal thing to do.

I found a case where the assertion itself does not hold:

> +Assert(crc_ok &&
> +       memcmp(dataDisk, &controlData, sizeof(ControlFileData)) == 0);

During XLOG_CHECKPOINT_ONLINE replay, xlog_redo() updates
ControlFile->checkPointCopy.nextXid under ControlFileLock without
calling UpdateControlFile(). The in-memory copy can therefore differ
from the on-disk file even while we hold the lock.

I can trigger this with the v13 patch on current master: pause a
standby's base backup at basebackup-before-send-files, allocate an XID
on the primary, run CHECKPOINT and wait for replay. Resuming the backup
hits this assertion in backup_control_file().

IIUC, this mismatch does not mean that pg_control is torn. Perhaps we
could drop the whole-struct equality assertion?

On the motivation, we have had many WAL-G support requests from users
who removed backup_label to get a restored cluster to start. Here is
one example [0], and a pgBackRest case prompted by the startup hint [1].
I think protecting against this mistake is worthwhile.

One concern about describing pg_resetwal as "the only way to clear it
otherwise", even with the corruption warning: a WAL-G user actually
removed backup_label and ran pg_resetwal, thinking this only meant
giving up PITR [2]. Could we frame this as bypassing the protection,
not as a way to clear the flag? It cannot replace the WAL replay needed
to make a backup consistent.

Thank you!


Best regards, Andrey Borodin.

[0] https://github.com/wal-g/wal-g/issues/115
[1] https://github.com/pgbackrest/pgbackrest/issues/839#issuecomment-532253964
[2] https://github.com/wal-g/wal-g/issues/1279#issuecomment-1144554054






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-15 18:45  David Steele <david@pgbackrest.org>
  parent: Andrey Borodin <x4mmm@yandex-team.ru>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-09-15 18:45 UTC (permalink / raw)
  To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: shihao zhong <zhong950419@gmail.com>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi Andrey,

Thank you for having a look at the patch.

On 9/15/26 21:13, Andrey Borodin wrote:
> 
> On 15 Sep 2026, David Steele wrote:
>> I left this as is from v11. I think it's a pretty normal thing to do.
> 
> I found a case where the assertion itself does not hold:
> 
>> +Assert(crc_ok &&
>> +       memcmp(dataDisk, &controlData, sizeof(ControlFileData)) == 0);
> 
> During XLOG_CHECKPOINT_ONLINE replay, xlog_redo() updates
> ControlFile->checkPointCopy.nextXid under ControlFileLock without
> calling UpdateControlFile(). The in-memory copy can therefore differ
> from the on-disk file even while we hold the lock.
> 
> I can trigger this with the v13 patch on current master: pause a
> standby's base backup at basebackup-before-send-files, allocate an XID
> on the primary, run CHECKPOINT and wait for replay. Resuming the backup
> hits this assertion in backup_control_file().
> 
> IIUC, this mismatch does not mean that pg_control is torn. Perhaps we
> could drop the whole-struct equality assertion?

The mismatch is not an issue as far as I can see. I'll leave that 
determination up to Michael, though, since he added the assert.

> On the motivation, we have had many WAL-G support requests from users
> who removed backup_label to get a restored cluster to start. Here is
> one example [0], and a pgBackRest case prompted by the startup hint [1].
> I think protecting against this mistake is worthwhile.

I've seen this now more times than I can count, even from people that I 
would expect to know better. The hint from Postgres to delete 
backup_label certainly does not help the situation, though even with 
this patch we can't remove it completely. I'm hoping to at least deal 
with the common case.

> One concern about describing pg_resetwal as "the only way to clear it
> otherwise", even with the corruption warning: a WAL-G user actually
> removed backup_label and ran pg_resetwal, thinking this only meant
> giving up PITR [2]. Could we frame this as bypassing the protection,
> not as a way to clear the flag? It cannot replace the WAL replay needed
> to make a backup consistent.
Yeah, I have also seen pg_resetwal abused in various ways. I added that 
language based on feedback from Shihao and I think it should be 
documented but perhaps we need more cautionary language there. Let's see 
what people think.

Regards,
-David






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-16 04:14  shihao zhong <zhong950419@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: shihao zhong @ 2026-09-16 04:14 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi David, Andrey,

Tested v13 on master (04c4c1c3a96). recovery, pg_rewind, pg_basebackup and
regress all pass. Two comments.

1. The pg_resetwal sentence in pg_basebackup.sgml

I asked for the field to be documented and still think it should be, but the
wording offers pg_resetwal as a way out and it is not one. It clears the
flag
and resets the WAL, but leaves backup_label, so the cluster still does not
start. The flag is gone by then, so the hint v13 just conditioned away is
back, and the server itself says "try removing the file backup_label". They
do, and it starts silently inconsistent. Unpatched master behaves
identically,
what is new is that users get sent there. Say what clears the flag, but make
restoring backup_label the remedy. Also rename the "pg_resetwal clears the
flag" subtest in 057.
Separately, pg_resetwal never mentions backup_label in code or docs, and
always leaves an unstartable cluster when one is present. A warning there
would cut this off at the first step. Happy to post that as its own patch.

2. The assert in backup_control_file()

Michael, this one is for you since David left the call to you. Assert builds
only, so no production impact, but the invariant is false and I would drop
the block rather than narrow it. Only xlog_redo() for XLOG_CHECKPOINT_ONLINE
updates the struct without calling UpdateControlFile, so
checkPointCopy.nextXid
is the only field that can differ, and InitWalRecovery replaces
checkPointCopy
wholesale when backup_label is present. The copy we ship is correct either
way, and asserting it equals the file contradicts the premise that shared
memory is authoritative. One detail on Andrey's recipe: a checkpoint
replayed
before the backup never trips it, since do_pg_backup_start forces a
restartpoint. Only replay during file sending matters.

Otherwise patch 01 looks ready to me.

Thanks,
Shihao

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-26 14:58  David Steele <david@pgbackrest.org>
  parent: shihao zhong <zhong950419@gmail.com>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-09-26 14:58 UTC (permalink / raw)
  To: shihao zhong <zhong950419@gmail.com>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi Andrew and Shihao,

On 9/16/26 07:14, shihao zhong wrote:
> 
> Tested v13 on master (04c4c1c3a96). recovery, pg_rewind, pg_basebackup and
> regress all pass. Two comments.
> 
> 1. The pg_resetwal sentence in pg_basebackup.sgml
> 
> I asked for the field to be documented and still think it should be, but the
> wording offers pg_resetwal as a way out and it is not one.

Agreed, so added more cautionary language in v14.

> what is new is that users get sent there. Say what clears the flag, but make
> restoring backup_label the remedy. Also rename the "pg_resetwal clears the
> flag" subtest in 057.

I updated the comment but left the test message as is -- I think it 
works with the new comment.

> Separately, pg_resetwal never mentions backup_label in code or docs, and
> always leaves an unstartable cluster when one is present. A warning there
> would cut this off at the first step. Happy to post that as its own patch.

Yes, I think a separate patch would be best. Feel free to CC me so I can 
review.

> 2. The assert in backup_control_file()
> 
> Michael, this one is for you since David left the call to you. 

I went ahead and removed the assert since as Andrey showed it is 
demonstrably not correct. We'll see what Michael thinks.

Regards,
-David

Attachments:

pgcontrol-basebackup-flag-v14.patch (text/plain, 26.2K) download From f3b16071d4e66572ff8c703e8be38061fc5a1296 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Sat, 26 Sep 2026 14:08:52 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own, so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 13 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ doc/src/sgml/ref/pg_rewind.sgml | 4 + src/backend/access/transam/xlog.c | 31 ++++ src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/backup/basebackup.c | 15 +- src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/058_backup_label_required.pl | 165 ++++++++++++++++++ 18 files changed, 300 insertions(+), 15 deletions(-) create mode 100644 src/test/recovery/t/058_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..984984b8959 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + Backup software using this API should set the flag itself if it is able to + modify <filename>pg_control</filename>. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index e56c9a22c42..f7ce58479a9 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3644,6 +3644,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index 3117968d125..64501bb3aab 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,18 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is removed, recovery refuses to start + rather than silently producing an inconsistent cluster. Restoring + <filename>backup_label</filename> to the root of the backup allows + recovery to proceed, and the flag is cleared once recovery has read the + file. <xref linkend="app-pgresetwal"/> also clears the flag, but discarding + the WAL bypasses the replay that recovery needs, leaving an inconsistent + cluster. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index ac3d0c9328f..c4888c183fb 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. + <filename>pg_control</filename> is also flagged to require + <filename>backup_label</filename>, so that removing that file does not + allow the target to start up in an inconsistent state. The flag is + cleared once the target has processed <filename>backup_label</filename>. </para> </step> <step> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9ec0be77ca0..ca47b67dc02 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10589,6 +10589,37 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + * + * The caller's buffer must be at least PG_CONTROL_FILE_SIZE bytes and is + * zero-padded to that size. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * suitably aligned for ControlFileData. + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData.backupLabelRequired = true; + + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index fff8d57ac61..f61e4c0199b 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index e3c04ecd810..74ab7781512 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -340,9 +341,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -365,14 +366,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *) controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 41afc4c1316..9d36c5b4a84 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -922,6 +922,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 7a590b7e1ea..23f42bc7602 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -340,6 +340,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f46427258e3..19523bc882e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12442,9 +12442,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index ebb12dd8766..8fa08a99ee4 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -66,6 +66,7 @@ tests += { 't/055_cascade_reconnect.pl', 't/056_standby_snapshot_export.pl', 't/057_snapshot_commit_race.pl', + 't/058_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..6b02eb99881 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 058_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/058_backup_label_required.pl b/src/test/recovery/t/058_backup_label_required.pl new file mode 100644 index 00000000000..5d1cee27054 --- /dev/null +++ b/src/test/recovery/t/058_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal clears the flag. This is not ideal since it leaves an inconsistent +# cluster, but at least verify that the flag is cleared. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-27 16:02  shihao zhong <zhong950419@gmail.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: shihao zhong @ 2026-09-27 16:02 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi David,

v14 addresses both of my comments, thanks.

I'll post the pg_resetwal warning as its own thread and CC you.

Marking this Ready for Committer.

Thanks,
Shihao

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-27 16:13  shihao zhong <zhong950419@gmail.com>
  parent: shihao zhong <zhong950419@gmail.com>
  0 siblings, 1 reply; 64+ messages in thread

From: shihao zhong @ 2026-09-27 16:13 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi David,

On the pg_resetwal patch, one question before I write it. A restored backup
is never cleanly shut down, so pg_resetwal already needs -f there. A
backup_label check that -f overrides would only change the message. I'd
rather refuse even with -f, like the postmaster.pid check, and make the user
remove backup_label by hand. 058 would then need to remove backup_label
before its pg_resetwal step. Does that work for you? Or you would prefer
to have a warning message only.

Thanks,
Shihao

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-09-28 06:03  David Steele <david@pgbackrest.org>
  parent: shihao zhong <zhong950419@gmail.com>
  0 siblings, 0 replies; 64+ messages in thread

From: David Steele @ 2026-09-28 06:03 UTC (permalink / raw)
  To: shihao zhong <zhong950419@gmail.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andy Pogrebnoi <andrew.pogrebnoi@percona.com>; Michael Paquier <michael@paquier.xyz>; Haibo Yan <tristan.yim@gmail.com>; pgsql-hackers; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; Shaun Thomas <shaun.thomas@pgedge.com>

Hi Shihao,

On 9/27/26 19:13, shihao zhong wrote:
> 
> On the pg_resetwal patch, one question before I write it. A restored backup
> is never cleanly shut down, so pg_resetwal already needs -f there. A
> backup_label check that -f overrides would only change the message. I'd
> rather refuse even with -f, like the postmaster.pid check, and make the user
> remove backup_label by hand. 058 would then need to remove backup_label
> before its pg_resetwal step.
I would definitely prefer to force the user to remove backup_label 
before pg_resetwal will continue (even with -f). The presence of 
backup_label indicates they are making a mistake by using pg_resetwal, 
when they should probably be configuring recovery instead.

Regards,
-David






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-21 13:07  David Steele <david@pgbackrest.org>
  parent: solai v <solai.cdac@gmail.com>
  1 sibling, 1 reply; 64+ messages in thread

From: David Steele @ 2026-10-21 13:07 UTC (permalink / raw)
  To: pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

Patch has been rebased.

Attachments:

pgcontrol-basebackup-flag-v15.patch (text/plain, 26.2K) download From 988158b5d6cb9666a866ae5c5cc538a265f450f4 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Wed, 21 Oct 2026 13:02:21 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. Another advantage is that the copy of pg_control used by pg_basebackup is guaranteed not to be torn. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own, so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 13 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ doc/src/sgml/ref/pg_rewind.sgml | 4 + src/backend/access/transam/xlog.c | 31 ++++ src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/backup/basebackup.c | 15 +- src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/access/xlog.h | 1 + src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/061_backup_label_required.pl | 165 ++++++++++++++++++ 18 files changed, 300 insertions(+), 15 deletions(-) create mode 100644 src/test/recovery/t/061_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..984984b8959 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + Backup software using this API should set the flag itself if it is able to + modify <filename>pg_control</filename>. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index c9c0261b52f..2f0651c694e 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3651,6 +3651,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index b4a0faccf41..93676057394 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,18 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is removed, recovery refuses to start + rather than silently producing an inconsistent cluster. Restoring + <filename>backup_label</filename> to the root of the backup allows + recovery to proceed, and the flag is cleared once recovery has read the + file. <xref linkend="app-pgresetwal"/> also clears the flag, but discarding + the WAL bypasses the replay that recovery needs, leaving an inconsistent + cluster. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index ac3d0c9328f..c4888c183fb 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. + <filename>pg_control</filename> is also flagged to require + <filename>backup_label</filename>, so that removing that file does not + allow the target to start up in an inconsistent state. The flag is + cleared once the target has processed <filename>backup_label</filename>. </para> </step> <step> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 8636ad9ab2c..7e0a7e452d1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10608,6 +10608,37 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup and update it + * to require a backup label for recovery. Also recalculate the CRC. + * + * The caller's buffer must be at least PG_CONTROL_FILE_SIZE bytes and is + * zero-padded to that size. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * suitably aligned for ControlFileData. + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlData.backupLabelRequired = true; + + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 20be788da78..56368a9378f 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index d7c485dc002..c9ee9e2cac8 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -340,9 +341,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -365,14 +366,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *) controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 634d966da9e..1b9846fb8a4 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -924,6 +924,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 2bf5714b51e..a2a640178c4 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -343,6 +343,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1f21a8eb6d4..a885c173c25 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12443,9 +12443,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd07a7..31b0cbd34e1 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..8633b690cc6 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 061_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/061_backup_label_required.pl b/src/test/recovery/t/061_backup_label_required.pl new file mode 100644 index 00000000000..5d1cee27054 --- /dev/null +++ b/src/test/recovery/t/061_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal clears the flag. This is not ideal since it leaves an inconsistent +# cluster, but at least verify that the flag is cleared. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-23 12:42  David Steele <david@pgbackrest.org>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 2 replies; 64+ messages in thread

From: David Steele @ 2026-10-23 12:42 UTC (permalink / raw)
  To: pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 10/21/26 15:07, David Steele wrote:
> Patch has been rebased.

A committer pointed out to me that part of this patch fixes a bug and 
could be back-patched, so it should be separate.

Patch 01 now fixes only the possibility of a torn pg_control file. It 
makes the copy out of memory to reduce the time the lock is held, but it 
could alternately read the pg_control file in the lock as was done by 
Thomas in c558e6fd92f. If the consensus is that the latter approach is 
preferred I'm happy to make that change.

Patch 02 implements the flag to enforce the presence of backup_label and 
credits Andres with the original idea. This is the bulk of the prior patch.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-v16-01-fix-tear.patch (text/plain, 5.2K) download From 8a2822f52faa46955be97340741310327351374f Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Fri, 23 Oct 2026 12:33:19 +0000 Subject: Avoid a torn copy of pg_control in base backups. Base backups read pg_control from disk, which can return a torn copy if the server is rewriting the file at the same time. At least ext4 and ntfs can expose partially written contents to a concurrent read, and a backup containing a torn pg_control fails to start because the CRC does not match. Commit 63a582222c6 dealt with the same hazard in frontend tools by retrying the read until the CRC is stable, which is appropriate for programs that have not access to shared memory. Commit c558e6fd92f made the SQL functions take ControlFileLock while reading the file. A base backup runs in the server, which already has the contents of pg_control in shared memory, so instead copy them from there under ControlFileLock. No read of the file is needed at all, and the lock is held only for the copy. The CRC is recalculated because the in-memory copy can be updated without being written, e.g. during replay of an online checkpoint on a standby, so the CRC it holds cannot be relied on. Backups taken with the low-level API still copy pg_control from disk and are not helped by this change. --- src/backend/access/transam/xlog.c | 36 +++++++++++++++++++++++++++++++ src/backend/backup/basebackup.c | 15 ++++++------- src/include/access/xlog.h | 1 + 3 files changed, 43 insertions(+), 9 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 8636ad9ab2c..e89227ed8b9 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10608,6 +10608,42 @@ do_pg_abort_backup(int code, Datum arg) } } +/* + * Create a consistent copy of control data to be used for backup. + * + * Reading pg_control from disk while it is being rewritten can return a torn + * copy, so make the copy from shared memory under ControlFileLock instead. + * + * The caller's buffer must be at least PG_CONTROL_FILE_SIZE bytes and is + * zero-padded to that size. + * + * All field access is done through a local, properly-aligned ControlFileData; + * the caller's buffer is only ever written via memcpy() and so need not be + * suitably aligned for ControlFileData. + */ +void +backup_control_file(uint8 *controlFile) +{ + ControlFileData controlData; + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(&controlData, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + /* + * Recalculate the CRC. The in-memory copy can be updated without being + * written, e.g. during replay of an online checkpoint, in which case its + * CRC does not match its contents. + */ + INIT_CRC32C(controlData.crc); + COMP_CRC32C(controlData.crc, &controlData, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData.crc); + + /* Copy into the caller's buffer, zero-padded to the full file size */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, &controlData, sizeof(ControlFileData)); +} + /* * Register a handler that will warn about unterminated backups at end of * session, unless this has already been done. diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index d7c485dc002..c9ee9e2cac8 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,6 +23,7 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" @@ -340,9 +341,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + uint8 controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -365,14 +366,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendtblspclinks, &manifest, InvalidOid, ib); /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + backup_control_file(controlFile); + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *) controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 2bf5714b51e..a2a640178c4 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -343,6 +343,7 @@ extern void do_pg_backup_start(const char *backupidstr, bool fast, StringInfo tblspcmapfile); extern void do_pg_backup_stop(BackupState *state, bool waitforarchive); extern void do_pg_abort_backup(int code, Datum arg); +extern void backup_control_file(uint8 *controlFile); extern void register_persistent_abort_backup_handler(void); extern SessionBackupState get_backup_status(void); -- 2.34.1
pgcontrol-v16-02-backuplabel-flag.patch (text/plain, 23.7K) download From 54689b92d326f7e25897458d8b4b45562eeb7f66 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Fri, 23 Oct 2026 12:33:19 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own, so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Andres Freund suggested this approach in an earlier thread that proposed moving the contents of backup_label into pg_control. Control and catalog version bumps are required. Suggested-by: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20231120203746.x3dgftmwvfgl5tdy%40awork3.anarazel.de --- doc/src/sgml/backup.sgml | 13 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ doc/src/sgml/ref/pg_rewind.sgml | 4 + src/backend/access/transam/xlog.c | 5 +- src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/061_backup_label_required.pl | 165 ++++++++++++++++++ 16 files changed, 266 insertions(+), 7 deletions(-) create mode 100644 src/test/recovery/t/061_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..984984b8959 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + Backup software using this API should set the flag itself if it is able to + modify <filename>pg_control</filename>. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index c9c0261b52f..2f0651c694e 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3651,6 +3651,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index b4a0faccf41..93676057394 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,18 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is removed, recovery refuses to start + rather than silently producing an inconsistent cluster. Restoring + <filename>backup_label</filename> to the root of the backup allows + recovery to proceed, and the flag is cleared once recovery has read the + file. <xref linkend="app-pgresetwal"/> also clears the flag, but discarding + the WAL bypasses the replay that recovery needs, leaving an inconsistent + cluster. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index ac3d0c9328f..c4888c183fb 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. + <filename>pg_control</filename> is also flagged to require + <filename>backup_label</filename>, so that removing that file does not + allow the target to start up in an inconsistent state. The flag is + cleared once the target has processed <filename>backup_label</filename>. </para> </step> <step> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index e89227ed8b9..3ae2deb787e 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -10609,7 +10609,8 @@ do_pg_abort_backup(int code, Datum arg) } /* - * Create a consistent copy of control data to be used for backup. + * Create a consistent copy of control data to be used for backup and flag it + * to require backup_label for recovery. * * Reading pg_control from disk while it is being rewritten can return a torn * copy, so make the copy from shared memory under ControlFileLock instead. @@ -10630,6 +10631,8 @@ backup_control_file(uint8 *controlFile) memcpy(&controlData, ControlFile, sizeof(ControlFileData)); LWLockRelease(ControlFileLock); + controlData.backupLabelRequired = true; + /* * Recalculate the CRC. The in-memory copy can be updated without being * written, e.g. during replay of an online checkpoint, in which case its diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 20be788da78..56368a9378f 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 634d966da9e..1b9846fb8a4 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -924,6 +924,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1f21a8eb6d4..a885c173c25 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12443,9 +12443,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd07a7..31b0cbd34e1 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..8633b690cc6 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 061_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/061_backup_label_required.pl b/src/test/recovery/t/061_backup_label_required.pl new file mode 100644 index 00000000000..5d1cee27054 --- /dev/null +++ b/src/test/recovery/t/061_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal clears the flag. This is not ideal since it leaves an inconsistent +# cluster, but at least verify that the flag is cleared. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-23 15:47  Heikki Linnakangas <hlinnaka@iki.fi>
  parent: David Steele <david@pgbackrest.org>
  1 sibling, 2 replies; 64+ messages in thread

From: Heikki Linnakangas @ 2026-10-23 15:47 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 23/10/2026 15:42, David Steele wrote:
> On 10/21/26 15:07, David Steele wrote:
>> Patch has been rebased.
> 
> A committer pointed out to me that part of this patch fixes a bug and 
> could be back-patched, so it should be separate.
> 
> Patch 01 now fixes only the possibility of a torn pg_control file.

Looks good to me.

> It makes the copy out of memory to reduce the time the lock is held,
> but it could alternately read the pg_control file in the lock as was
> done by Thomas in c558e6fd92f. If the consensus is that the latter
> approach is preferred I'm happy to make that change.
+0.5 for following the example of c558e6fd92f. This isn't 
performance-sensitive so that aspect doesn't matter. But it seems a 
little surprising if a backup might have more up-to-date values of some 
fields than what the SQL functions show.

- Heikki







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-23 16:17  Heikki Linnakangas <hlinnaka@iki.fi>
  parent: David Steele <david@pgbackrest.org>
  1 sibling, 1 reply; 64+ messages in thread

From: Heikki Linnakangas @ 2026-10-23 16:17 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 23/10/2026 15:42, David Steele wrote:
> Patch 02 implements the flag to enforce the presence of backup_label and 
> credits Andres with the original idea. This is the bulk of the prior patch.
Makes sense at high level.

> --- a/doc/src/sgml/backup.sgml
> +++ b/doc/src/sgml/backup.sgml
> @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true);
>       vital to the backup working and must be written byte for byte without
>       modification, which may require opening the file in binary mode.
>      </para>
> +    <para>
> +     Note that a backup made with the low-level API copies
> +     <filename>pg_control</filename> from the running cluster, so it does not
> +     contain the flag that <xref linkend="app-pgbasebackup"/> sets to make
> +     <filename>backup_label</filename> mandatory for recovery. Removing
> +     <filename>backup_label</filename> from such a backup will not prevent the
> +     cluster from starting; it will instead recover from the checkpoint
> +     recorded in <filename>pg_control</filename>. Unless that happens to be the
> +     checkpoint the backup started from, recovery will begin too late and
> +     silently produce an inconsistent cluster.
> +     Backup software using this API should set the flag itself if it is able to
> +     modify <filename>pg_control</filename>.
> +    </para>
>     </listitem>
>     <listitem>
>      <para>

That's good advice, but the text feels awkward. Which flag do you need 
to set exactly? And maybe the text should be placed elsewhere; the 
copying of the pg_control file happened earlier already and you should 
probably modify it already when you copy it.

> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
> index b4a0faccf41..93676057394 100644
> --- a/doc/src/sgml/ref/pg_basebackup.sgml
> +++ b/doc/src/sgml/ref/pg_basebackup.sgml
> @@ -60,6 +60,18 @@ PostgreSQL documentation
>     <xref linkend="app-pgdump"/> must be used.
>    </para>
>  
> +  <para>
> +   The copy of <filename>pg_control</filename> stored in the backup is flagged
> +   to require the <filename>backup_label</filename> file for recovery. If
> +   <filename>backup_label</filename> is removed, recovery refuses to start
> +   rather than silently producing an inconsistent cluster. Restoring
> +   <filename>backup_label</filename> to the root of the backup allows
> +   recovery to proceed, and the flag is cleared once recovery has read the
> +   file. <xref linkend="app-pgresetwal"/> also clears the flag, but discarding
> +   the WAL bypasses the replay that recovery needs, leaving an inconsistent
> +   cluster.
> +  </para>
> +

You shouldn't try to restore a backup_label file into an existing backup 
that's missing it, so that wording sounds dangerous.

>    <para>
>     The backup is made over a regular <productname>PostgreSQL</productname>
>     connection that uses the replication protocol. The connection must be made
> diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml
> index ac3d0c9328f..c4888c183fb 100644
> --- a/doc/src/sgml/ref/pg_rewind.sgml
> +++ b/doc/src/sgml/ref/pg_rewind.sgml
> @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b
>        defined as the result of <literal>pg_current_wal_insert_lsn()</literal>
>        when rewinding from a live source or the last checkpoint LSN when
>        rewinding from a stopped source.
> +      <filename>pg_control</filename> is also flagged to require
> +      <filename>backup_label</filename>, so that removing that file does not
> +      allow the target to start up in an inconsistent state. The flag is
> +      cleared once the target has processed <filename>backup_label</filename>.
>       </para>
>      </step>
>      <step>

That doesn't seem worth mentioning here; too much detail.

> diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
> index 20be788da78..56368a9378f 100644
> --- a/src/backend/access/transam/xlogrecovery.c
> +++ b/src/backend/access/transam/xlogrecovery.c
> @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr,
>  					ereport(FATAL,
>  							errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X",
>  								   LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)),
> +							ControlFile->backupLabelRequired ?
> +							errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n"
> +									"Do not remove \"%s/backup_label\"; it is required to recover this backup.",
> +									DataDir, DataDir, DataDir) :
>  							errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n"
>  									"If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n"
>  									"Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.",

Umm, not new with his patch, but why do we still have that hint to 
remove backup_label? Is that ever the right thing to do? Why didn't we 
remove that hint when exclusive-mode backups were removed?

Idea for further hardening: If you attempt to start up postgres and the 
backup_label is present, set the flag in the control file. That way, 
even if the backup tool doesn't modify the control file, it will be set 
on first startup attempt. If you delete the backup_label later, you 
still won't be able to start up into an inconsistent state. It won't 
protect you if you remove the backup_label before attempting to start 
up, but it'd catch some cases.

- Heikki







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-24 05:25  Michael Paquier <michael@paquier.xyz>
  parent: Heikki Linnakangas <hlinnaka@iki.fi>
  1 sibling, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-10-24 05:25 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On Fri, Oct 23, 2026 at 06:47:24PM +0300, Heikki Linnakangas wrote:
> Looks good to me.

Apologies for not following closely what you have been doing lately in
this area, David.  I should have followed, but well..

+void
+backup_control_file(uint8 *controlFile)
+{
+    ControlFileData controlData;
+
+    LWLockAcquire(ControlFileLock, LW_SHARED);
+    memcpy(&controlData, ControlFile, sizeof(ControlFileData));
+    LWLockRelease(ControlFileLock);
[...]
                 /* ... and pg_control after everything else. */
-                if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0)
-                    ereport(ERROR,
-                            (errcode_for_file_access(),
-                             errmsg("could not stat file \"%s\": %m",
-                                    XLOG_CONTROL_FILE)));
-                sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf,
-                         false, InvalidOid, InvalidOid,
-                         InvalidRelFileNumber, 0, &manifest, 0, NULL, 0);
+                backup_control_file(controlFile);
+                sendFileWithContent(sink, XLOG_CONTROL_FILE,
+                                    (char *) controlFile, PG_CONTROL_FILE_SIZE,
+                                    &manifest);

Yep, that's clearly something that should be backpatched, at quick
glance.  Heikki, are you planning to take care of this part?
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-24 17:14  David Steele <david@pgbackrest.org>
  parent: Heikki Linnakangas <hlinnaka@iki.fi>
  1 sibling, 0 replies; 64+ messages in thread

From: David Steele @ 2026-10-24 17:14 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 10/23/26 17:47, Heikki Linnakangas wrote:
> On 23/10/2026 15:42, David Steele wrote:
> 
>> It makes the copy out of memory to reduce the time the lock is held,
>> but it could alternately read the pg_control file in the lock as was
>> done by Thomas in c558e6fd92f. If the consensus is that the latter
>> approach is preferred I'm happy to make that change.
> +0.5 for following the example of c558e6fd92f. This isn't performance- 
> sensitive so that aspect doesn't matter. But it seems a little 
> surprising if a backup might have more up-to-date values of some fields 
> than what the SQL functions show.
Another +0.5 from me makes it +1. We know that pg_control in memory can 
be ahead of pg_control on disk (for standbys) and we are quite sure that 
is not an issue but it seems better not to take the risk for something 
that we are going to back patch. Also better to use the same mechanism 
everywhere.

I've attached a patch 01 that does that and rebased 02 on it. 02 does 
not make any of the changes that you requested in your later message but 
I'll take care of those separately. Since minor releases are coming up 
it seems better to prioritize 01.

Note that back patching this below PG17 will require also back patching 
7606175991f (see commit message for details).

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-v17-01-fix-tear.patch (text/plain, 4.0K) download From c0a88ffad9ee92757e6ef2e7a806b8d73b77da5f Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Sat, 24 Oct 2026 15:59:45 +0000 Subject: Avoid a torn copy of pg_control in base backups. Base backups read pg_control from disk without any protection against concurrent writes by the server. At least ext4 and NTFS can expose partially written contents to a concurrent read and a backup containing a torn pg_control fails to start because the CRC does not match. Read pg_control while holding ControlFileLock, as commit c558e6fd92f did for the SQL functions that read pg_control. The file is read into memory and sent from there so the lock is not held while the contents are streamed to the client. Commit 63a582222c6 dealt with the same hazard in frontend tools by retrying the read until the CRC is stable, but a base backup runs in the server and can use ControlFileLock instead. Since the read is done under the lock, a CRC mismatch means that pg_control is actually corrupt so raise an error rather than include it in the backup. Backups taken with the low-level API still copy pg_control from disk and are not affected by this change. Back-patch to all supported branches. Before v17, sendFileWithContent() gets the length of the content with strlen(), which would truncate pg_control at its first zero byte. Commit 7606175991f, which added the length argument, must be back-patched first. It is confined to basebackup.c where the function is static. --- src/backend/backup/basebackup.c | 35 ++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index d7c485dc002..b5740f758dd 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,9 +23,11 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" +#include "common/controldata_utils.h" #include "common/file_perm.h" #include "common/file_utils.h" #include "lib/stringinfo.h" @@ -42,6 +44,7 @@ #include "storage/checksum.h" #include "storage/dsm_impl.h" #include "storage/ipc.h" +#include "storage/lwlock.h" #include "storage/reinit.h" #include "utils/builtins.h" #include "utils/guc.h" @@ -340,9 +343,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + ControlFileData *controlData; + bool crc_ok; + char controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -364,15 +369,27 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid, ib); - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) + /* + * ... and pg_control after everything else. Read it while + * holding ControlFileLock so that a concurrent write cannot + * produce a torn copy. + */ + LWLockAcquire(ControlFileLock, LW_SHARED); + controlData = get_controlfile(DataDir, &crc_ok); + LWLockRelease(ControlFileLock); + + if (!crc_ok) ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + (errmsg("calculated CRC checksum does not match value stored in file"))); + + /* Send the full file, zero-padded as it is on disk */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, controlData, sizeof(ControlFileData)); + pfree(controlData); + + sendFileWithContent(sink, XLOG_CONTROL_FILE, + controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { -- 2.34.1
pgcontrol-v17-02-backuplabel-flag.patch (text/plain, 23.6K) download From a715fdb1ec2990a151ce456adb9d9b3ae1d79cd9 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Sat, 24 Oct 2026 15:59:45 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Andres Freund suggested this approach in an earlier thread that proposed moving the contents of backup_label into pg_control. Control and catalog version bumps are required. Suggested-by: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20231120203746.x3dgftmwvfgl5tdy%40awork3.anarazel.de --- doc/src/sgml/backup.sgml | 13 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ doc/src/sgml/ref/pg_rewind.sgml | 4 + src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/backup/basebackup.c | 8 + src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/061_backup_label_required.pl | 165 ++++++++++++++++++ 16 files changed, 270 insertions(+), 6 deletions(-) create mode 100644 src/test/recovery/t/061_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..984984b8959 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + Backup software using this API should set the flag itself if it is able to + modify <filename>pg_control</filename>. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index c9c0261b52f..2f0651c694e 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3651,6 +3651,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index b4a0faccf41..93676057394 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,18 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is removed, recovery refuses to start + rather than silently producing an inconsistent cluster. Restoring + <filename>backup_label</filename> to the root of the backup allows + recovery to proceed, and the flag is cleared once recovery has read the + file. <xref linkend="app-pgresetwal"/> also clears the flag, but discarding + the WAL bypasses the replay that recovery needs, leaving an inconsistent + cluster. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index ac3d0c9328f..c4888c183fb 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -455,6 +455,10 @@ GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint, b defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. + <filename>pg_control</filename> is also flagged to require + <filename>backup_label</filename>, so that removing that file does not + allow the target to start up in an inconsistent state. The flag is + cleared once the target has processed <filename>backup_label</filename>. </para> </step> <step> diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 20be788da78..56368a9378f 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup_label file that was created during the backup.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index b5740f758dd..688f65c9405 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -382,6 +382,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, ereport(ERROR, (errmsg("calculated CRC checksum does not match value stored in file"))); + /* Require backup_label for recovery from this backup */ + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlData, + offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); + /* Send the full file, zero-padded as it is on disk */ memset(controlFile, 0, PG_CONTROL_FILE_SIZE); memcpy(controlFile, controlData, sizeof(ControlFileData)); diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 634d966da9e..1b9846fb8a4 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -924,6 +924,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1f21a8eb6d4..a885c173c25 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12443,9 +12443,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd07a7..31b0cbd34e1 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..8633b690cc6 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 061_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/061_backup_label_required.pl b/src/test/recovery/t/061_backup_label_required.pl new file mode 100644 index 00000000000..5d1cee27054 --- /dev/null +++ b/src/test/recovery/t/061_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal clears the flag. This is not ideal since it leaves an inconsistent +# cluster, but at least verify that the flag is cleared. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-26 11:42  David Steele <david@pgbackrest.org>
  parent: Heikki Linnakangas <hlinnaka@iki.fi>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-10-26 11:42 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 10/23/26 18:17, Heikki Linnakangas wrote:
> On 23/10/2026 15:42, David Steele wrote:
>> Patch 02 implements the flag to enforce the presence of backup_label 
>> and credits Andres with the original idea. This is the bulk of the 
>> prior patch.
> Makes sense at high level.
> 
>> --- a/doc/src/sgml/backup.sgml
>> +++ b/doc/src/sgml/backup.sgml
>> @@ -1031,6 +1031,19 @@ SELECT * FROM pg_backup_stop(wait_for_archive 
>> => true);
>>       vital to the backup working and must be written byte for byte 
>> without
>>       modification, which may require opening the file in binary mode.
>>      </para>
>> +    <para>
>> +     Note that a backup made with the low-level API copies
>> +     <filename>pg_control</filename> from the running cluster, so it 
>> does not
>> +     contain the flag that <xref linkend="app-pgbasebackup"/> sets to 
>> make
>> +     <filename>backup_label</filename> mandatory for recovery. Removing
>> +     <filename>backup_label</filename> from such a backup will not 
>> prevent the
>> +     cluster from starting; it will instead recover from the checkpoint
>> +     recorded in <filename>pg_control</filename>. Unless that happens 
>> to be the
>> +     checkpoint the backup started from, recovery will begin too late 
>> and
>> +     silently produce an inconsistent cluster.
>> +     Backup software using this API should set the flag itself if it 
>> is able to
>> +     modify <filename>pg_control</filename>.
>> +    </para>
>>     </listitem>
>>     <listitem>
>>      <para>
> 
> That's good advice, but the text feels awkward. Which flag do you need 
> to set exactly? And maybe the text should be placed elsewhere; the 
> copying of the pg_control file happened earlier already and you should 
> probably modify it already when you copy it.

On second thought I think this should just be removed rather than moved. 
Modifying pg_control is not something we talk about in the docs, so 
there are no good examples, and I don't think we should start here. 
Removed in v18.

>> +   The copy of <filename>pg_control</filename> stored in the backup 
>> is flagged
>> +   to require the <filename>backup_label</filename> file for 
>> recovery. If
>> +   <filename>backup_label</filename> is removed, recovery refuses to 
>> start
>> +   rather than silently producing an inconsistent cluster. Restoring
>> +   <filename>backup_label</filename> to the root of the backup allows
>> +   recovery to proceed, and the flag is cleared once recovery has 
>> read the
>> +   file. <xref linkend="app-pgresetwal"/> also clears the flag, but 
>> discarding
>> +   the WAL bypasses the replay that recovery needs, leaving an 
>> inconsistent
>> +   cluster.
> 
> You shouldn't try to restore a backup_label file into an existing backup 
> that's missing it, so that wording sounds dangerous.

Agreed, I updated this in v18 to recommend a fresh restored including 
backup label. Who knows where a user might get a backup_label from 
otherwise?

I also updated the related hint to suggest a new restore:

-					errhint("Restore the backup_label file that was created during the 
backup."));
+					errhint("Restore the backup again, including the backup_label 
file."));

>>        defined as the result of <literal>pg_current_wal_insert_lsn()</ 
>> literal>
>>        when rewinding from a live source or the last checkpoint LSN when
>>        rewinding from a stopped source.
>> +      <filename>pg_control</filename> is also flagged to require
>> +      <filename>backup_label</filename>, so that removing that file 
>> does not
>> +      allow the target to start up in an inconsistent state. The flag is
>> +      cleared once the target has processed <filename>backup_label</ 
>> filename>.
> 
> That doesn't seem worth mentioning here; too much detail.

Agreed. Removed in v18.

>> @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, 
>> bool *wasShutdown_ptr,
>>                      ereport(FATAL,
>>                              errmsg("could not find redo location %X/ 
>> %08X referenced by checkpoint record at %X/%08X",
>>                                     LSN_FORMAT_ARGS(checkPoint.redo), 
>> LSN_FORMAT_ARGS(CheckPointLoc)),
>> +                            ControlFile->backupLabelRequired ?
>> +                            errhint("Touch \"%s/recovery.signal\" or 
>> \"%s/standby.signal\" and add required recovery options.\n"
>> +                                    "Do not remove \"%s/ 
>> backup_label\"; it is required to recover this backup.",
>> +                                    DataDir, DataDir, DataDir) :
>>                              errhint("If you are restoring from a 
>> backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add 
>> required recovery options.\n"
>>                                      "If you are not restoring from a 
>> backup, try removing the file \"%s/backup_label\".\n"
>>                                      "Be careful: removing \"%s/ 
>> backup_label\" will result in a corrupt cluster if restoring from a 
>> backup.",
> 
> Umm, not new with his patch, but why do we still have that hint to 
> remove backup_label? Is that ever the right thing to do? Why didn't we 
> remove that hint when exclusive-mode backups were removed?

Unfortunately I believe there is still a use case for this hint. When 
making a backup, e.g. via snapshot, it is still easier to write the 
backup_label into PGDATA, make the snapshot, then remove it. If the 
cluster crashes before the removal then the answer is to remove the 
backup label, just as it was if exclusive backup crashed.

I'm not excited about the above use case but not sure we can discount it 
entirely.

What we could do is document that the above use case should be 
accomplished by copying backup_label to another file in PGDATA, e.g. 
backup_label.snap, and then renaming it when the snapshot is restored. I 
think this is reasonable given that recovery.signal/standby.signal still 
needed to be created so there is some work to be done on the restore 
side. Maybe then it would be OK to remove the hint?

That feels like the subject for a separate patch, though. I have added 
it to my notes.

> Idea for further hardening: If you attempt to start up postgres and the 
> backup_label is present, set the flag in the control file. That way, 
> even if the backup tool doesn't modify the control file, it will be set 
> on first startup attempt. If you delete the backup_label later, you 
> still won't be able to start up into an inconsistent state. It won't 
> protect you if you remove the backup_label before attempting to start 
> up, but it'd catch some cases.
This would be dangerous if we accept that the above use case is valid. 
In that case on restart the cluster would be bricked where in fact 
removing the backup_label is the right thing to do.

Maybe we could set the flag when backup_label is present *and* either 
standby.signal or recovery.signal. That works for the case above.

I originally had a patch that allowed backup software to get a non-torn 
copy of pg_control with the flag set from pg_backup_stop(). If I can get 
these patches committed I plan to reintroduce it. I think that is a 
better answer for third-party software.

If we do decide to go with your idea instead (or in addition to), I'd 
prefer to pursue it in a separate patch. I have added it to my notes.

Regards,
-David

Attachments: download all (2 files, .tar.gz)

pgcontrol-v18-01-fix-tear.patch (text/plain, 4.0K) download From 463d1e1689a8a5fb1a70e75d0b2600027d1bbfe6 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Mon, 26 Oct 2026 11:31:05 +0000 Subject: Avoid a torn copy of pg_control in base backups. Base backups read pg_control from disk without any protection against concurrent writes by the server. At least ext4 and NTFS can expose partially written contents to a concurrent read and a backup containing a torn pg_control fails to start because the CRC does not match. Read pg_control while holding ControlFileLock, as commit c558e6fd92f did for the SQL functions that read pg_control. The file is read into memory and sent from there so the lock is not held while the contents are streamed to the client. Commit 63a582222c6 dealt with the same hazard in frontend tools by retrying the read until the CRC is stable, but a base backup runs in the server and can use ControlFileLock instead. Since the read is done under the lock, a CRC mismatch means that pg_control is actually corrupt so raise an error rather than include it in the backup. Backups taken with the low-level API still copy pg_control from disk and are not affected by this change. Back-patch to all supported branches. Before v17, sendFileWithContent() gets the length of the content with strlen(), which would truncate pg_control at its first zero byte. Commit 7606175991f, which added the length argument, must be back-patched first. It is confined to basebackup.c where the function is static. --- src/backend/backup/basebackup.c | 35 ++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index d7c485dc002..b5740f758dd 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -23,9 +23,11 @@ #include "backup/basebackup_incremental.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "catalog/pg_tablespace_d.h" #include "commands/defrem.h" #include "common/compression.h" +#include "common/controldata_utils.h" #include "common/file_perm.h" #include "common/file_utils.h" #include "lib/stringinfo.h" @@ -42,6 +44,7 @@ #include "storage/checksum.h" #include "storage/dsm_impl.h" #include "storage/ipc.h" +#include "storage/lwlock.h" #include "storage/reinit.h" #include "utils/builtins.h" #include "utils/guc.h" @@ -340,9 +343,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; char *backup_label; + ControlFileData *controlData; + bool crc_ok; + char controlFile[PG_CONTROL_FILE_SIZE]; bbsink_begin_archive(sink, "base.tar"); @@ -364,15 +369,27 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid, ib); - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) + /* + * ... and pg_control after everything else. Read it while + * holding ControlFileLock so that a concurrent write cannot + * produce a torn copy. + */ + LWLockAcquire(ControlFileLock, LW_SHARED); + controlData = get_controlfile(DataDir, &crc_ok); + LWLockRelease(ControlFileLock); + + if (!crc_ok) ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest, 0, NULL, 0); + (errmsg("calculated CRC checksum does not match value stored in file"))); + + /* Send the full file, zero-padded as it is on disk */ + memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + memcpy(controlFile, controlData, sizeof(ControlFileData)); + pfree(controlData); + + sendFileWithContent(sink, XLOG_CONTROL_FILE, + controlFile, PG_CONTROL_FILE_SIZE, + &manifest); } else { -- 2.34.1
pgcontrol-v18-02-backuplabel-flag.patch (text/plain, 22.6K) download From 919b94a3367a260aa563f15bbd37683676749796 Mon Sep 17 00:00:00 2001 From: David Steele <david@pgbackrest.org> Date: Mon, 26 Oct 2026 11:31:05 +0000 Subject: Add pg_control flag to prevent recovery without backup_label. Harden recovery by adding a flag to pg_control to indicate that backup_label is required. This prevents the user from deleting backup_label resulting in an inconsistent recovery. Recovery hints that suggest removing backup_label are omitted when the flag is set, since in that case pg_control shows that the data directory came from a backup and removing backup_label would leave it unable to start at all. This functionality is limited to pg_basebackup and pg_rewind (or any software comfortable with modifying pg_control). pg_rewind creates a backup_label of its own so the flag is set there as well, which means that removing backup_label after a rewind no longer allows the cluster to start. Andres Freund suggested this approach in an earlier thread that proposed moving the contents of backup_label into pg_control. Control and catalog version bumps are required. Suggested-by: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/20231120203746.x3dgftmwvfgl5tdy%40awork3.anarazel.de --- doc/src/sgml/backup.sgml | 11 ++ doc/src/sgml/func/func-info.sgml | 5 + doc/src/sgml/ref/pg_basebackup.sgml | 12 ++ src/backend/access/transam/xlogrecovery.c | 27 ++- src/backend/backup/basebackup.c | 8 + src/backend/utils/misc/pg_controldata.c | 7 +- src/bin/pg_controldata/pg_controldata.c | 2 + src/bin/pg_resetwal/pg_resetwal.c | 1 + src/bin/pg_rewind/pg_rewind.c | 1 + src/bin/pg_rewind/t/RewindTest.pm | 12 ++ src/include/catalog/pg_control.h | 4 + src/include/catalog/pg_proc.dat | 6 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/042_low_level_backup.pl | 8 + .../recovery/t/061_backup_label_required.pl | 165 ++++++++++++++++++ 15 files changed, 264 insertions(+), 6 deletions(-) create mode 100644 src/test/recovery/t/061_backup_label_required.pl diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 168444eccc5..5bba58ad8e8 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -1031,6 +1031,17 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode. </para> + <para> + Note that a backup made with the low-level API copies + <filename>pg_control</filename> from the running cluster, so it does not + contain the flag that <xref linkend="app-pgbasebackup"/> sets to make + <filename>backup_label</filename> mandatory for recovery. Removing + <filename>backup_label</filename> from such a backup will not prevent the + cluster from starting; it will instead recover from the checkpoint + recorded in <filename>pg_control</filename>. Unless that happens to be the + checkpoint the backup started from, recovery will begin too late and + silently produce an inconsistent cluster. + </para> </listitem> <listitem> <para> diff --git a/doc/src/sgml/func/func-info.sgml b/doc/src/sgml/func/func-info.sgml index c9c0261b52f..2f0651c694e 100644 --- a/doc/src/sgml/func/func-info.sgml +++ b/doc/src/sgml/func/func-info.sgml @@ -3651,6 +3651,11 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres} <entry><type>boolean</type></entry> </row> + <row> + <entry><structfield>backup_label_required</structfield></entry> + <entry><type>boolean</type></entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index b4a0faccf41..a94ee549524 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml +++ b/doc/src/sgml/ref/pg_basebackup.sgml @@ -60,6 +60,18 @@ PostgreSQL documentation <xref linkend="app-pgdump"/> must be used. </para> + <para> + The copy of <filename>pg_control</filename> stored in the backup is flagged + to require the <filename>backup_label</filename> file for recovery. If + <filename>backup_label</filename> is missing, recovery refuses to start + rather than silently producing an inconsistent cluster. The backup must + then be restored again, including <filename>backup_label</filename>. The + flag is cleared once recovery has read <filename>backup_label</filename>. + <xref linkend="app-pgresetwal"/> also clears the flag, but discarding the + WAL bypasses the replay that recovery needs, leaving an inconsistent + cluster. + </para> + <para> The backup is made over a regular <productname>PostgreSQL</productname> connection that uses the replication protocol. The connection must be made diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 20be788da78..21cb1ce2797 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -604,6 +604,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not find redo location %X/%08X referenced by checkpoint record at %X/%08X", LSN_FORMAT_ARGS(checkPoint.redo), LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -615,6 +619,10 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, ereport(FATAL, errmsg("could not locate required checkpoint record at %X/%08X", LSN_FORMAT_ARGS(CheckPointLoc)), + ControlFile->backupLabelRequired ? + errhint("Touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" + "Do not remove \"%s/backup_label\"; it is required to recover this backup.", + DataDir, DataDir, DataDir) : errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", @@ -659,7 +667,14 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, } else { - /* No backup_label file has been found if we are here. */ + /* + * No backup_label file has been found if we are here. Error if the + * control file requires backup_label. + */ + if (ControlFile->backupLabelRequired) + ereport(FATAL, + errmsg("could not find backup_label required for recovery"), + errhint("Restore the backup again, including the backup_label file.")); /* * If tablespace_map file is present without backup_label file, there @@ -939,11 +954,21 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. + * + * backupLabelRequired is set to false since backup_label is no longer + * required once pg_control has been updated on disk. If recovery + * terminates abnormally between when pg_control is updated and + * backup_label is renamed then on restart pg_control will be + * reinitialized from backup_label. If the user manually deletes + * backup_label before restarting then recovery will proceed with the + * contents of pg_control just as it would if the crash had happened + * directly after backup_label rename. */ if (haveBackupLabel) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; + ControlFile->backupLabelRequired = false; if (backupFromStandby) { diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index b5740f758dd..688f65c9405 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -382,6 +382,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, ereport(ERROR, (errmsg("calculated CRC checksum does not match value stored in file"))); + /* Require backup_label for recovery from this backup */ + controlData->backupLabelRequired = true; + + INIT_CRC32C(controlData->crc); + COMP_CRC32C(controlData->crc, controlData, + offsetof(ControlFileData, crc)); + FIN_CRC32C(controlData->crc); + /* Send the full file, zero-padded as it is on disk */ memset(controlFile, 0, PG_CONTROL_FILE_SIZE); memcpy(controlFile, controlData, sizeof(ControlFileData)); diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 1f47302fe2a..232c888d594 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -168,8 +168,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[6]; + bool nulls[6]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -201,6 +201,9 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[4] = BoolGetDatum(ControlFile->backupEndRequired); nulls[4] = false; + values[5] = BoolGetDatum(ControlFile->backupLabelRequired); + nulls[5] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..8bcca89733b 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -305,6 +305,8 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); + printf(_("Backup label required: %s\n"), + ControlFile->backupLabelRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), wal_level_str(ControlFile->wal_level)); printf(_("wal_log_hints setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 634d966da9e..1b9846fb8a4 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -924,6 +924,7 @@ RewriteControlFile(void) ControlFile.backupStartPoint = InvalidXLogRecPtr; ControlFile.backupEndPoint = InvalidXLogRecPtr; ControlFile.backupEndRequired = false; + ControlFile.backupLabelRequired = false; /* * The old WAL is gone and the new position may lie below the old diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d2521dab333..a16081669e1 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -783,6 +783,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupLabelRequired = true; /* * Keep the target's own data checksum state. Most of the data directory diff --git a/src/bin/pg_rewind/t/RewindTest.pm b/src/bin/pg_rewind/t/RewindTest.pm index 32aeca80f13..b2a7fca9d02 100644 --- a/src/bin/pg_rewind/t/RewindTest.pm +++ b/src/bin/pg_rewind/t/RewindTest.pm @@ -353,6 +353,18 @@ sub run_pg_rewind croak("Incorrect test mode specified"); } + # pg_rewind writes a backup_label of its own when it rewinds, and the + # target must then require it for recovery; the flag is cleared again when + # the target is restarted below and recovery consumes backup_label. When + # the target is already an ancestor of the source no rewind takes place, + # and pg_control must be left alone so that the target still starts. + my $label_required = + -f "$primary_pgdata/backup_label" ? 'yes' : 'no'; + command_like( + [ 'pg_controldata', '--pgdata' => $primary_pgdata ], + qr/Backup label required: +$label_required/, + 'backup_label requirement matches whether a rewind took place'); + # Now move back postgresql.conf with old settings move( "$tmp_folder/primary-postgresql.conf.tmp", diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..f83928e2ca3 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -172,12 +172,16 @@ typedef struct ControlFileData * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. + * + * If backupLabelRequired is true, then a backup_label file must be + * present in order for recovery to proceed. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; + bool backupLabelRequired; /* * Parameter settings that determine if the WAL can be used for archival diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 1f21a8eb6d4..a885c173c25 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12443,9 +12443,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool,bool}', + proargmodes => '{o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required,backup_label_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd07a7..31b0cbd34e1 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_backup_label_required.pl', ], }, } diff --git a/src/test/recovery/t/042_low_level_backup.pl b/src/test/recovery/t/042_low_level_backup.pl index 7ed54e611eb..8633b690cc6 100644 --- a/src/test/recovery/t/042_low_level_backup.pl +++ b/src/test/recovery/t/042_low_level_backup.pl @@ -165,4 +165,12 @@ ok( $node_replica->log_contains( 'FATAL: .*could not locate required checkpoint record at'), 'ends with FATAL for missing required checkpoint record'); +# A low-level backup copies pg_control from the running cluster, so it does not +# require backup_label and the hint still offers removing the file as a way to +# recover a cluster that is not being restored from a backup. Backups made +# with pg_basebackup do require it and get a different hint; see +# 061_backup_label_required.pl. +ok($node_replica->log_contains('try removing the file .*backup_label'), + 'hint offers removing backup_label when it is not required'); + done_testing(); diff --git a/src/test/recovery/t/061_backup_label_required.pl b/src/test/recovery/t/061_backup_label_required.pl new file mode 100644 index 00000000000..5d1cee27054 --- /dev/null +++ b/src/test/recovery/t/061_backup_label_required.pl @@ -0,0 +1,165 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +# Test the pg_control flag that makes backup_label mandatory for recovery. +# +# pg_basebackup stores a modified copy of pg_control in the backup, with a flag +# set that makes recovery refuse to start if backup_label is missing. This +# prevents the silent corruption that results from removing the file, both for +# backups taken from a primary and from a standby. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::RecursiveCopy; +use PostgreSQL::Test::Utils; +use Test::More; + +# Rename backup_label in the data directory of the given node, start it, and +# check that recovery refuses to proceed. Then put the file back. +sub check_startup_without_backup_label +{ + my ($node, $test_name) = @_; + my $data_dir = $node->data_dir; + my $log_offset = -s $node->logfile; + + rename("$data_dir/backup_label", "$data_dir/backup_label.tmp") + or BAIL_OUT("could not rename $data_dir/backup_label"); + + is($node->start(fail_ok => 1), 0, "$test_name: startup fails"); + ok( $node->log_contains( + 'FATAL: .*could not find backup_label required for recovery', + $log_offset), + "$test_name: ends with FATAL for missing backup_label"); + + rename("$data_dir/backup_label.tmp", "$data_dir/backup_label") + or BAIL_OUT("could not rename $data_dir/backup_label.tmp"); + return; +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); + +# Set wal_keep_size to prevent WAL segment recycling after the checkpoints +# enforced by the backups below. The standby is created from the first backup +# but started only after later backups have advanced the primary, so it must +# still be able to stream the segments in between. This is set before the +# first backup so that it is inherited by the standbys. +$node_primary->append_conf('postgresql.conf', 'wal_keep_size = 64MB'); +$node_primary->start; + +$node_primary->safe_psql('postgres', + 'CREATE TABLE tab_int AS SELECT generate_series(1, 1000) AS a'); + +# Take a backup from the primary. The copy of pg_control stored in the backup +# must require backup_label, while the control file of the running cluster is +# left alone. +my $backup_name = 'backup_primary'; +$node_primary->backup($backup_name); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_primary->backup_dir . '/' . $backup_name + ], + qr/Backup label required: +yes/, + 'backup taken from a primary requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $node_primary->data_dir ], + qr/Backup label required: +no/, + 'control file of the source cluster is unchanged'); +is( $node_primary->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'pg_control_recovery() reports the flag not set on the source cluster'); + +# pg_resetwal clears the flag. This is not ideal since it leaves an inconsistent +# cluster, but at least verify that the flag is cleared. +my $reset_dir = $node_primary->backup_dir . '/' . $backup_name . '_reset'; +PostgreSQL::Test::RecursiveCopy::copypath( + $node_primary->backup_dir . '/' . $backup_name, $reset_dir); +chmod(0700, $reset_dir) or BAIL_OUT("could not chmod $reset_dir"); + +command_ok([ 'pg_resetwal', '--force', '--pgdata' => $reset_dir ], + 'pg_resetwal runs on a backup that requires backup_label'); +command_like( + [ 'pg_controldata', '--pgdata' => $reset_dir ], + qr/Backup label required: +no/, + 'pg_resetwal clears the flag'); + +# Restoring that backup without backup_label must not start. +my $node_restored = PostgreSQL::Test::Cluster->new('restored'); +$node_restored->init_from_backup($node_primary, $backup_name); + +check_startup_without_backup_label($node_restored, 'backup from primary'); + +# With backup_label back in place recovery completes, and the flag is cleared +# so that subsequent restarts no longer need the file. +$node_restored->start; +is($node_restored->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'restored cluster has the expected contents'); +is( $node_restored->safe_psql( + 'postgres', 'SELECT backup_label_required FROM pg_control_recovery()'), + 'f', + 'flag is cleared once recovery has completed'); +$node_restored->stop; + +command_like( + [ 'pg_controldata', '--pgdata' => $node_restored->data_dir ], + qr/Backup label required: +no/, + 'control file no longer requires backup_label after recovery'); + +# A backup made without WAL cannot reach a consistent state on its own, and +# recovery reports that it could not locate the checkpoint record. Since +# pg_control requires backup_label, the hint for that error must not repeat the +# usual advice to remove the file. +my $nowal_backup = 'backup_nowal'; +$node_primary->backup($nowal_backup, + backup_options => [ '--wal-method' => 'none' ]); + +my $node_nowal = PostgreSQL::Test::Cluster->new('nowal'); +$node_nowal->init_from_backup($node_primary, $nowal_backup); + +my $nowal_offset = -s $node_nowal->logfile; +is($node_nowal->start(fail_ok => 1), 0, + 'backup without WAL fails to start when no recovery options are set'); +ok( $node_nowal->log_contains( + 'FATAL: .*could not locate required checkpoint record', $nowal_offset), + 'missing checkpoint record is reported'); +ok( $node_nowal->log_contains('HINT: .*recovery\.signal', $nowal_offset), + 'hint points at the recovery signal files'); +ok( $node_nowal->log_contains('Do not remove .*backup_label', $nowal_offset), + 'hint tells the user to keep backup_label'); +ok( !$node_nowal->log_contains('try removing the file', $nowal_offset), + 'hint does not suggest removing backup_label'); + +# A backup taken from a standby gets the same treatment. This is the case that +# previously required backup software to copy pg_control last. +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup($node_primary, $backup_name, + has_streaming => 1); +$node_standby->start; +$node_primary->wait_for_replay_catchup($node_standby); + +my $standby_backup = 'backup_standby'; +$node_standby->backup($standby_backup); + +command_like( + [ + 'pg_controldata', + '--pgdata' => $node_standby->backup_dir . '/' . $standby_backup + ], + qr/Backup label required: +yes/, + 'backup taken from a standby requires backup_label'); + +my $node_standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$node_standby2->init_from_backup($node_standby, $standby_backup, + has_streaming => 1); + +check_startup_without_backup_label($node_standby2, 'backup from standby'); + +$node_standby2->start; +$node_standby->wait_for_replay_catchup($node_standby2, $node_primary); +is($node_standby2->safe_psql('postgres', 'SELECT count(*) FROM tab_int'), + 1000, 'cascading standby from a standby backup is caught up'); + +done_testing(); -- 2.34.1
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 00:00  Michael Paquier <michael@paquier.xyz>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-10-28 00:00 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On Sat, Oct 24, 2026 at 02:25:30PM +0900, Michael Paquier wrote:
>                  /* ... and pg_control after everything else. */
> -                if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0)
> -                    ereport(ERROR,
> -                            (errcode_for_file_access(),
> -                             errmsg("could not stat file \"%s\": %m",
> -                                    XLOG_CONTROL_FILE)));
> -                sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf,
> -                         false, InvalidOid, InvalidOid,
> -                         InvalidRelFileNumber, 0, &manifest, 0, NULL, 0);
> +                backup_control_file(controlFile);
> +                sendFileWithContent(sink, XLOG_CONTROL_FILE,
> +                                    (char *) controlFile, PG_CONTROL_FILE_SIZE,
> +                                    &manifest);
> 
> Yep, that's clearly something that should be backpatched, at quick
> glance.  Heikki, are you planning to take care of this part?

Hearing nothing, I have added that into my own pipeline and applies
the fix down to v15.  v14 has been left out, the branch going EOL soon
with a problem that is extremely going to happen in practice does not
give me a warm feeling.  For v15 and v16, I have backpatched
d2c1e18de9d1 first.  The fix has been applied as of 4a92c8c29bd0.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 06:38  Michael Paquier <michael@paquier.xyz>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 2 replies; 64+ messages in thread

From: Michael Paquier @ 2026-10-28 06:38 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On Wed, Oct 28, 2026 at 09:00:44AM +0900, Michael Paquier wrote:
> Hearing nothing, I have added that into my own pipeline and applies
> the fix down to v15.  v14 has been left out, the branch going EOL soon
> with a problem that is extremely going to happen in practice does not
> give me a warm feeling.  For v15 and v16, I have backpatched
> d2c1e18de9d1 first.  The fix has been applied as of 4a92c8c29bd0.

Based on a sudden report from billbug, under Solaris 11, this gets
fun:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=billbug&dt=2026-10-28%2004%3A00%3A01

And we have a crash in a BASE_BACKUP command:
2026-10-28 05:23:30.548 CET [1429:4] LOG:  client backend (PID 1498)
was terminated by signal 11: Segmentation Fault
2026-10-28 05:23:30.548 CET [1429:5] DETAIL:  Failed process was
running: BASE_BACKUP ( LABEL 'pg_basebackup base backup',  PROGRESS,
CHECKPOINT 'fast',  WAIT 0,  TABLESPACE_MAP,  MANIFEST 'yes',  TARGET
'client')

This runs under valgrind with -DRELCACHE_FORCE_RELEASE, and of course
I am not able to reproduce it with combinations of these three.

The closest thing that I can see in recent history would be
040932c31146.  One possible explanation is PG_CONTROL_FILE_SIZE added
to the stack of perform_base_backup()?  I'm still puzzled without more
information from valgrind, but if that's right we could try a
palloc0() instead.

My blind suggestion of fix would be the attached, that I cannot really
test.  Thoughts or comments?
--
Michael

Attachments:

basebackup-controlfile.patch (text/plain, 1.0K) download diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 0534ac217830..f8c3520e5b8e 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -347,7 +347,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, char *backup_label; ControlFileData *controlData; bool crc_ok; - char controlFile[PG_CONTROL_FILE_SIZE]; + char *controlFile; bbsink_begin_archive(sink, "base.tar"); @@ -379,13 +379,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink, (errmsg("calculated CRC checksum does not match value stored in file"))); /* Send the full file, zero-padded as it is on disk */ - memset(controlFile, 0, PG_CONTROL_FILE_SIZE); + controlFile = palloc0(PG_CONTROL_FILE_SIZE); memcpy(controlFile, controlData, sizeof(ControlFileData)); pfree(controlData); sendFileWithContent(sink, XLOG_CONTROL_FILE, controlFile, PG_CONTROL_FILE_SIZE, &manifest); + pfree(controlFile); } else {
signature.asc (application/pgp-signature, 832B) download ^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 08:56  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 2 replies; 64+ messages in thread

From: David Steele @ 2026-10-28 08:56 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>; hofstetter@jomasoft.ch

On 10/28/26 07:38, Michael Paquier wrote:
> On Wed, Oct 28, 2026 at 09:00:44AM +0900, Michael Paquier wrote:
>> Hearing nothing, I have added that into my own pipeline and applies
>> the fix down to v15.  v14 has been left out, the branch going EOL soon
>> with a problem that is extremely going to happen in practice does not
>> give me a warm feeling.  For v15 and v16, I have backpatched
>> d2c1e18de9d1 first.  The fix has been applied as of 4a92c8c29bd0.
> 
> Based on a sudden report from billbug, under Solaris 11, this gets
> fun:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=billbug&dt=2026-10-28%2004%3A00%3A01
> 
> And we have a crash in a BASE_BACKUP command:
> 2026-10-28 05:23:30.548 CET [1429:4] LOG:  client backend (PID 1498)
> was terminated by signal 11: Segmentation Fault
> 2026-10-28 05:23:30.548 CET [1429:5] DETAIL:  Failed process was
> running: BASE_BACKUP ( LABEL 'pg_basebackup base backup',  PROGRESS,
> CHECKPOINT 'fast',  WAIT 0,  TABLESPACE_MAP,  MANIFEST 'yes',  TARGET
> 'client')
> 
> This runs under valgrind with -DRELCACHE_FORCE_RELEASE, and of course
> I am not able to reproduce it with combinations of these three.

Are you sure valgrind is running? I'm not too familiar with build farm 
configuration but it set 'use_valgrind' => undef, which seems to imply 
it is off even if -DUSE_VALGRIND is also set.

> 
> The closest thing that I can see in recent history would be
> 040932c31146.  One possible explanation is PG_CONTROL_FILE_SIZE added
> to the stack of perform_base_backup()?  I'm still puzzled without more
> information from valgrind, but if that's right we could try a
> palloc0() instead.

This was my first thought, but I can't see why 8KiB on the stack would 
be an issue. To me this looks a lot like a latent issue being exposed 
because we've pushed things around on the stack a bit.

> My blind suggestion of fix would be the attached, that I cannot really
> test.  Thoughts or comments?

The patch looks good to me but I'm concerned it would just be masking 
the real issue even if it works.

I'd be curious to see if the back branches run green when they run. It 
would also be helpful to get a stack trace.

I've CC'd billbug's owner. The HEAD run with the 2026-10-28 04:00 UTC
snapshot crashed a client backend (PID 1498) running BASE_BACKUP with
SIGSEGV during src/bin/pg_basebackup/t/010_pg_basebackup.pl, but no
backtrace made it into the logs. Could you check whether a core file was
written and get a backtrace from it? If no core was written then the
coreadm settings may be redirecting or suppressing it. Any additional 
info you can provide would be greatly appreciated.

Regards,
-David






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 09:48  Heikki Linnakangas <hlinnaka@iki.fi>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Heikki Linnakangas @ 2026-10-28 09:48 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 26/10/2026 13:42, David Steele wrote:
> On 10/23/26 18:17, Heikki Linnakangas wrote:
>> On 23/10/2026 15:42, David Steele wrote:
>> Umm, not new with his patch, but why do we still have that hint to 
>> remove backup_label? Is that ever the right thing to do? Why didn't we 
>> remove that hint when exclusive-mode backups were removed?
> 
> Unfortunately I believe there is still a use case for this hint. When 
> making a backup, e.g. via snapshot, it is still easier to write the 
> backup_label into PGDATA, make the snapshot, then remove it. If the 
> cluster crashes before the removal then the answer is to remove the 
> backup label, just as it was if exclusive backup crashed.
> 
> I'm not excited about the above use case but not sure we can discount it 
> entirely.

If you take an atomic filesystem-level snapshot, restoring it is 
equivalent to crash recovery, so you don't need a backup_label to 
restore it to a consistent state. There's no need for pg_start_backup() 
or a backup_label in that case. Admittedly, a non-atomic snapshot is a 
different story.

But I don't see how you could do that anyway. The backup_label is 
returned by pg_stop_backup(). If the sequence is:

1. pg_start_backup()
2. take filesystem snapshot
3. pg_stop_backup()

there's no way to include the backup_label in the snapshot, because you 
don't have it until you call pg_stop_backup().

- Heikki







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 10:55  David Steele <david@pgbackrest.org>
  parent: Heikki Linnakangas <hlinnaka@iki.fi>
  0 siblings, 0 replies; 64+ messages in thread

From: David Steele @ 2026-10-28 10:55 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; +Cc: Michael Paquier <michael@paquier.xyz>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On 10/28/26 10:48, Heikki Linnakangas wrote:
> On 26/10/2026 13:42, David Steele wrote:
>> On 10/23/26 18:17, Heikki Linnakangas wrote:
>>> On 23/10/2026 15:42, David Steele wrote:
>>> Umm, not new with his patch, but why do we still have that hint to 
>>> remove backup_label? Is that ever the right thing to do? Why didn't 
>>> we remove that hint when exclusive-mode backups were removed?
>>
>> Unfortunately I believe there is still a use case for this hint. When 
>> making a backup, e.g. via snapshot, it is still easier to write the 
>> backup_label into PGDATA, make the snapshot, then remove it. If the 
>> cluster crashes before the removal then the answer is to remove the 
>> backup label, just as it was if exclusive backup crashed.
>>
>> I'm not excited about the above use case but not sure we can discount 
>> it entirely.
> 
> If you take an atomic filesystem-level snapshot, restoring it is 
> equivalent to crash recovery, so you don't need a backup_label to 
> restore it to a consistent state. There's no need for pg_start_backup() 
> or a backup_label in that case. Admittedly, a non-atomic snapshot is a 
> different story.

True, but I recommend *always* using the documented backup procedure, 
especially if PITR will be used on restore.

> But I don't see how you could do that anyway. The backup_label is 
> returned by pg_stop_backup(). If the sequence is:
> 
> 1. pg_start_backup()
> 2. take filesystem snapshot
> 3. pg_stop_backup()
> 
> there's no way to include the backup_label in the snapshot, because you 
> don't have it until you call pg_stop_backup().
Oops, yeah, that was an old pattern what was still usable up until PG14 
but should no longer be possible.

I searched the thread where we removed exclusive backup and I can't see 
any reason why we would have kept it, so it was probably an oversight.

Once Michael and I have resolved the build farm issue with 01 I'll 
rebase and make the removal a small patch before adding the pgcontrol 
flag -- unless you would prefer a separate thread.

Regards,
-David





^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 10:56  Michael Paquier <michael@paquier.xyz>
  parent: David Steele <david@pgbackrest.org>
  1 sibling, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-10-28 10:56 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>; hofstetter@jomasoft.ch, Noah Misch <noah@leadboat.com>

On Wed, Oct 28, 2026 at 08:56:25AM +0000, David Steele wrote:
> I've CC'd billbug's owner. The HEAD run with the 2026-10-28 04:00 UTC
> snapshot crashed a client backend (PID 1498) running BASE_BACKUP with
> SIGSEGV during src/bin/pg_basebackup/t/010_pg_basebackup.pl, but no
> backtrace made it into the logs. Could you check whether a core file was
> written and get a backtrace from it? If no core was written then the
> coreadm settings may be redirecting or suppressing it. Any additional info
> you can provide would be greatly appreciated.

We have what looks like a second report on HEAD and v19, different
pattern:
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=thorntail&dt=2026-10-28%2006%3A31%...

From 001's primary logs:
2026-10-28 10:44:13.113 MSK [1706208:4] LOG:  client backend (PID
1706227) was terminated by signal 10: Bus error
2026-10-28 10:44:13.113 MSK [1706208:5] DETAIL:  Failed process was
running: BASE_BACKUP (CHECKPOINT 'fast', MAX_RATE 32);

If something else is going on, I'm not quite sure what it is yet.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 15:32  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 64+ messages in thread

From: David Steele @ 2026-10-28 15:32 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>; hofstetter@jomasoft.ch, Noah Misch <noah@leadboat.com>

On 10/28/26 11:56, Michael Paquier wrote:
> On Wed, Oct 28, 2026 at 08:56:25AM +0000, David Steele wrote:
>> I've CC'd billbug's owner. The HEAD run with the 2026-10-28 04:00 UTC
>> snapshot crashed a client backend (PID 1498) running BASE_BACKUP with
>> SIGSEGV during src/bin/pg_basebackup/t/010_pg_basebackup.pl, but no
>> backtrace made it into the logs. Could you check whether a core file was
>> written and get a backtrace from it? If no core was written then the
>> coreadm settings may be redirecting or suppressing it. Any additional info
>> you can provide would be greatly appreciated.
> 
> We have what looks like a second report on HEAD and v19, different
> pattern:
> https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=thorntail&dt=2026-10-28%2006%3A31%...
> 
>  From 001's primary logs:
> 2026-10-28 10:44:13.113 MSK [1706208:4] LOG:  client backend (PID
> 1706227) was terminated by signal 10: Bus error
> 2026-10-28 10:44:13.113 MSK [1706208:5] DETAIL:  Failed process was
> running: BASE_BACKUP (CHECKPOINT 'fast', MAX_RATE 32);
> 
> If something else is going on, I'm not quite sure what it is yet.
thorntail is also SPARC so this failure narrows things down quite a bit.

The test that crashed is "base backup cleanly canceled". This test is 
designed to be cancelled in the middle so it is unlikely that it ever 
got to the new code, which isolates the extra 8KiB on the stack as the 
likely cause. Well, that already seemed clear but this is additional 
confirmation.

On the cancel path the only runtime difference from the prior code is 
the size of perform_base_backup()'s frame. The new 8kB buffer is 
declared inside PG_ENSURE_ERROR_CLEANUP() so the error from the cancel 
siglongjmp()s back into the larger frame. A bus error on SPARC is 
usually an alignment trap, so my guess is that the larger frame exposes 
an alignment problem somewhere in the unwinding. That fits with my 
earlier suspicion that this is a latent issue exposed by moving things 
around on the stack.

Since v19 is failing as well this is not specific to master. I think we 
should go ahead with your palloc0() patch to get the buildfarm green 
again. It takes the buffer out of the frame entirely. It may well be 
masking something, but that is better investigated with a backtrace than 
with a red buildfarm. Perhaps Noah can get one from thorntail.

[This analysis was aided by Claude but I double-checked the details.]

Regards,
-David







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-28 16:26  Noah Misch <noah@leadboat.com>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Noah Misch @ 2026-10-28 16:26 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>; hofstetter@jomasoft.ch

On Wed, Oct 28, 2026 at 03:32:34PM +0000, David Steele wrote:
> On 10/28/26 11:56, Michael Paquier wrote:
> > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=thorntail&dt=2026-10-28%2006%3A31%...

> A bus error on SPARC is usually an alignment trap

I agree.  SPARC Linux is one of the few platforms still doing those.

> Since v19 is failing as well this is not specific to master. I think we
> should go ahead with your palloc0() patch to get the buildfarm green again.
> It takes the buffer out of the frame entirely. It may well be masking
> something, but that is better investigated with a backtrace than with a red
> buildfarm. Perhaps Noah can get one from thorntail.

I won't be able to help with that directly, but:

- thorntail is cfarm202.cfarm.net; accounts are available:
  https://portal.cfarm.net/users/new/

- If a newer buildfarm client would collect a stack trace here, I'm happy to
  upgrade.






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 00:48  Michael Paquier <michael@paquier.xyz>
  parent: Noah Misch <noah@leadboat.com>
  0 siblings, 0 replies; 64+ messages in thread

From: Michael Paquier @ 2026-10-29 00:48 UTC (permalink / raw)
  To: Noah Misch <noah@leadboat.com>; +Cc: David Steele <david@pgbackrest.org>; Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>; hofstetter@jomasoft.ch

On Wed, Oct 28, 2026 at 09:26:41AM -0700, Noah Misch wrote:
> I won't be able to help with that directly, but:
> 
> - thorntail is cfarm202.cfarm.net; accounts are available:
>   https://portal.cfarm.net/users/new/
> 
> - If a newer buildfarm client would collect a stack trace here, I'm happy to
>   upgrade.

Could it be possible to send the result of an objdump -d on
perform_base_backup() perhaps, before and after the fact?  That should
offer the glimpse of a hint, at least.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 00:50  Michael Paquier <michael@paquier.xyz>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 1 reply; 64+ messages in thread

From: Michael Paquier @ 2026-10-29 00:50 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On Wed, Oct 28, 2026 at 03:38:55PM +0900, Michael Paquier wrote:
> My blind suggestion of fix would be the attached, that I cannot really
> test.  Thoughts or comments?

I unfortunately do not have a better idea after sleeping on it, so I
have applied this palloc0/pfree solution.  Now I'll go light a candle
to make the buildfarm gods kinder to me.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 08:14  Michael Paquier <michael@paquier.xyz>
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 64+ messages in thread

From: Michael Paquier @ 2026-10-29 08:14 UTC (permalink / raw)
  To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: David Steele <david@pgbackrest.org>; pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

On Thu, Oct 29, 2026 at 09:50:46AM +0900, Michael Paquier wrote:
> I unfortunately do not have a better idea after sleeping on it, so I
> have applied this palloc0/pfree solution.  Now I'll go light a candle
> to make the buildfarm gods kinder to me.

billbug has just switched back to green on HEAD after two successive
failures, so it looks like the stack sizing was the right guess:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=billbug&dt=2026-10-29%2004%3A00%3A03

So something is going on here.  How could we rule out if this is
something on the Postgres side with SPARC?
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 08:27  Marcel Hofstetter <hofstetter@jomasoft.ch>
  parent: David Steele <david@pgbackrest.org>
  1 sibling, 0 replies; 64+ messages in thread

From: Marcel Hofstetter @ 2026-10-29 08:27 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; Michael Paquier <michael@paquier.xyz>; Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: pgsql-hackers; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Fujii Masao <masao.fujii@gmail.com>; solai v <solai.cdac@gmail.com>

Am 28.10.2026 um 09:56 schrieb David Steele:
> On 10/28/26 07:38, Michael Paquier wrote:
>> On Wed, Oct 28, 2026 at 09:00:44AM +0900, Michael Paquier wrote:
>>> Hearing nothing, I have added that into my own pipeline and applies
>>> the fix down to v15.  v14 has been left out, the branch going EOL soon
>>> with a problem that is extremely going to happen in practice does not
>>> give me a warm feeling.  For v15 and v16, I have backpatched
>>> d2c1e18de9d1 first.  The fix has been applied as of 4a92c8c29bd0.
>>
>> Based on a sudden report from billbug, under Solaris 11, this gets
>> fun:
>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl? 
>> nm=billbug&dt=2026-10-28%2004%3A00%3A01
>>
>> And we have a crash in a BASE_BACKUP command:
>> 2026-10-28 05:23:30.548 CET [1429:4] LOG:  client backend (PID 1498)
>> was terminated by signal 11: Segmentation Fault
>> 2026-10-28 05:23:30.548 CET [1429:5] DETAIL:  Failed process was
>> running: BASE_BACKUP ( LABEL 'pg_basebackup base backup',  PROGRESS,
>> CHECKPOINT 'fast',  WAIT 0,  TABLESPACE_MAP,  MANIFEST 'yes',  TARGET
>> 'client')
>>
>> This runs under valgrind with -DRELCACHE_FORCE_RELEASE, and of course
>> I am not able to reproduce it with combinations of these three.
> 
> Are you sure valgrind is running? I'm not too familiar with build farm 
> configuration but it set 'use_valgrind' => undef, which seems to imply 
> it is off even if -DUSE_VALGRIND is also set.
> 
>>
>> The closest thing that I can see in recent history would be
>> 040932c31146.  One possible explanation is PG_CONTROL_FILE_SIZE added
>> to the stack of perform_base_backup()?  I'm still puzzled without more
>> information from valgrind, but if that's right we could try a
>> palloc0() instead.
> 
> This was my first thought, but I can't see why 8KiB on the stack would 
> be an issue. To me this looks a lot like a latent issue being exposed 
> because we've pushed things around on the stack a bit.
> 
>> My blind suggestion of fix would be the attached, that I cannot really
>> test.  Thoughts or comments?
> 
> The patch looks good to me but I'm concerned it would just be masking 
> the real issue even if it works.
> 
> I'd be curious to see if the back branches run green when they run. It 
> would also be helpful to get a stack trace.
> 
> I've CC'd billbug's owner. The HEAD run with the 2026-10-28 04:00 UTC
> snapshot crashed a client backend (PID 1498) running BASE_BACKUP with
> SIGSEGV during src/bin/pg_basebackup/t/010_pg_basebackup.pl, but no
> backtrace made it into the logs. Could you check whether a core file was
> written and get a backtrace from it? If no core was written then the
> coreadm settings may be redirecting or suppressing it. Any additional 
> info you can provide would be greatly appreciated.

sorry. core was disabled. enabled now for the future.
interesting: my second solaris11/sparc animal is fine.
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=margay&br=master

Best regards,
Marcel







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 18:11  Burd, Greg <gregburd@amazon.com>
  0 siblings, 1 reply; 64+ messages in thread

From: Burd, Greg @ 2026-10-29 18:11 UTC (permalink / raw)
  To: pgsql-hackers; michael@paquier.xyz <michael@paquier.xyz>; +Cc: hlinnaka@iki.fi <hlinnaka@iki.fi>; david@pgbackrest.org <david@pgbackrest.org>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>

On Thu, 29 Oct 2026 09:50:46 +0900, Michael Paquier <michael@paquier.xyz> wrote:
> On Wed, Oct 28, 2026 at 03:38:55PM +0900, Michael Paquier wrote:
> > My blind suggestion of fix would be the attached, that I cannot really
> > test.  Thoughts or comments?
> 
> I unfortunately do not have a better idea after sleeping on it, so I
> have applied this palloc0/pfree solution.  Now I'll go light a candle
> to make the buildfarm gods kinder to me.
> --
> Michael

Hey Michael,

As requested (back channel) I spent some quality time with my
UltraSPARC 45 this morning and dug into this.

TL;DR: it's a GCC bug, and c97931c16d fixes it for us by keeping
perform_base_backup()'s frame under 4kB.

I reproduced it on an UltraSPARC-IIIi running illumos with gcc 13.4,
-O2 and cassert, at 4a92c8c29b.  Your suggested reproduction works: a
file with a 114-byte name in PGDATA, then BASE_BACKUP through psql
with "replication=1", gives the SIGSEGV every time.  So does
cancelling a BASE_BACKUP (MAX_RATE 32), which is thorntail's case.
The core (mdb, full output attached):

  status: process terminated by SIGSEGV, addr=fffffffe7f74733c
  %pc = perform_base_backup+0x1c0:   st  %o0, [%g1]
  %npc = perform_base_backup+0x1480
  %i0 = 0x00000001008b6000
  perform_base_backup+0x1c0(...)
  SendBaseBackup+0x5a0(...)
  exec_replication_command+0x5b4(...)
  PostgresMain+0x8f0(...)

That store is in the delay slot of the branch right after
sigsetjmp() in PG_TRY(), on its second return.  Here is the
disassembly you asked for, at that spot:

  227de22095 (before), frame 2224 bytes:
    call  sigsetjmp
    ...
    bne,pn  %icc, <catch>
    st    %o0, [ %fp + 0x57 ]

  4a92c8c29b, frame 10368 bytes:
    sethi %hi(0x1c00), %i0         ! in the prologue
    ...
    call  sigsetjmp
    xor   %i0, -837, %g1
    add   %g1, %fp, %g1
    bne,pn  %icc, <catch>
    st    %o0, [ %g1 ]

  c97931c16d (fix), frame 2096 bytes: same as before.

With the 8kB array, the frame slots are more than 4kB from %fp, too
far for SPARC's 13-bit immediate offset, so GCC needs a register to
hold the offset.  It keeps the constant 0x1c00 in %i0 across the
sigsetjmp() call and reuses it afterwards, and later in the PG_TRY
block it loads a string address into %i0.  On SPARC, siglongjmp()
doesn't restore %l and %i registers from the jmp_buf; it resumes the
register window from the frame's save area, so %i0 comes back with
the value it had at siglongjmp() time.  In the core, %i0 is the
string address instead of 0x1c00, and (0x1008b6000 ^ -837) + %fp is
exactly the faulting address.  On other platforms jmp_buf saves the
callee-saved registers, so the constant would come back intact.  The
garbage address explains SIGSEGV on billbug and the occasional
SIGBUS on thorntail, and why the WARNING from do_pg_abort_backup()
is never logged: the backend dies before the catch block runs.

The GCC RTL dumps show where it goes wrong.  LRA does reload the
constant after the call, (set (reg %i0) (const_int 7168)), but
postreload's reload_cse_move2add() turns that into a no-op move,
because it thinks %i0 survives the call.  cselib resets its state at
a REG_SETJMP call, but move2add has no such check, on GCC master as
well.

To rule out Postgres entirely, the attached sj.c/sj2.c is a small
program with no Postgres code: a function with an 8kB local array and
a sigsetjmp().  With gcc 13.4 -O2 it segfaults on this machine, and
it does the same when built with Debian's sparc64-linux-gnu-gcc-13
and run under qemu.  It runs fine at -O1 and -O0, and with gcc 4.4.
I'm not sure if this has been, or should be, reported to GCC as yet.

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs.  With it reverted, the same TAP run
logs the SIGSEGV.  The test still reports PASS in that case because
it only checks that pg_basebackup fails.

I don't see a better fix on our side.  A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB.  In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]).  Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation.  I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

best.

-greg

Attachments:

  sj.c (application/octet-stream, 1.3K) download

  sj2.c (application/octet-stream, 461B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-29 23:08  Michael Paquier <michael@paquier.xyz>
  parent: Burd, Greg <gregburd@amazon.com>
  0 siblings, 2 replies; 64+ messages in thread

From: Michael Paquier @ 2026-10-29 23:08 UTC (permalink / raw)
  To: Burd, Greg <gregburd@amazon.com>; +Cc: pgsql-hackers; hlinnaka@iki.fi <hlinnaka@iki.fi>; david@pgbackrest.org <david@pgbackrest.org>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>

On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:
> Validation of c97931c16d on the same machine: with it applied, three
> runs each of both reproductions give the WARNING and ERROR and no
> crash, and t/010_pg_basebackup.pl passes all 168 tests with no
> backend crash in the server logs.  With it reverted, the same TAP run
> logs the SIGSEGV.  The test still reports PASS in that case because
> it only checks that pg_basebackup fails.

Ah, thanks for the in-depth investigation.  So the previous guess was
really right.  Candles work.

> I don't see a better fix on our side.  A helper function with
> the array would also work, but palloc0() is simpler, and both rely on
> the same thing: no PG_TRY() frame larger than 4kB.  In a core build
> on this machine, the only other function with a frame that large that
> calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]).  Its code
> after sigsetjmp() reloads the constant, so it is fine today, but only
> by luck of register allocation.  I haven't checked contrib or other
> GCC versions or as mentioned earlier against their bug system.

Ugh.  Even if the buffer itself is not used within the try/catch
block itself?  Getting entirely rid of the TRY/CATCH would lead to a
much nicer result overall: we'd be able to keep the local buffer.  Or
perhaps go to the simpler solution of a palloc/pfree?  I cannot
believe that this extra allocation would matter for lo_export() calls.

If you are proved to be right, it sounds to me that we ought to
fix and backpatch that as well, especially if there is nothing we can
do on the Postgres side.  I don't believe in luck and magic when it
comes to software programming.

Thoughts?
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-30 08:33  David Steele <david@pgbackrest.org>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 1 reply; 64+ messages in thread

From: David Steele @ 2026-10-30 08:33 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; Burd, Greg <gregburd@amazon.com>; +Cc: pgsql-hackers; hlinnaka@iki.fi <hlinnaka@iki.fi>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>

On 10/30/26 00:08, Michael Paquier wrote:
> On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:
>> Validation of c97931c16d on the same machine: with it applied, three
>> runs each of both reproductions give the WARNING and ERROR and no
>> crash, and t/010_pg_basebackup.pl passes all 168 tests with no
>> backend crash in the server logs.  With it reverted, the same TAP run
>> logs the SIGSEGV.  The test still reports PASS in that case because
>> it only checks that pg_basebackup fails.
> 
> Ah, thanks for the in-depth investigation.  So the previous guess was
> really right.  Candles work.

Thank you, Greg! This is exactly the confirmation we needed.

I would encourage you to file a bug with gcc. I've found a few bugs and 
the gcc folks are really easy to work with. Since you have a repro it 
will be easy to produce the pre-processed output they want.

One bad thing -- even if they fix the bug, distros may choose not to 
back patch it (so it may already be fixed). I had to maintain a patch 
for a gcc bug on ppc64le for years before the affected distros EOL'd. 
That bug limited where I could use const, which was pretty annoying.

>> I don't see a better fix on our side.  A helper function with
>> the array would also work, but palloc0() is simpler, and both rely on
>> the same thing: no PG_TRY() frame larger than 4kB.  In a core build
>> on this machine, the only other function with a frame that large that
>> calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]).  Its code
>> after sigsetjmp() reloads the constant, so it is fine today, but only
>> by luck of register allocation.  I haven't checked contrib or other
>> GCC versions or as mentioned earlier against their bug system.
> 
> Ugh.  Even if the buffer itself is not used within the try/catch
> block itself?  

Apparently so -- the failure of the base backup cancel test showed that. 
The buffer was never used and just the larger frame caused the problem.

> Getting entirely rid of the TRY/CATCH would lead to a
> much nicer result overall: we'd be able to keep the local buffer.  Or
> perhaps go to the simpler solution of a palloc/pfree?  I cannot
> believe that this extra allocation would matter for lo_export() calls.

palloc/pfree seems like the simpler solution.

> If you are proved to be right, it sounds to me that we ought to
> fix and backpatch that as well, especially if there is nothing we can
> do on the Postgres side.  I don't believe in luck and magic when it
> comes to software programming.
I've attached a patch for be_lo_export(). Working around compiler bugs 
is never fun but I'm not sure what choice we have.

Regards,
-David

Attachments:

lo-export-heap-buffer.patch (text/plain, 1005B) download diff --git a/src/backend/libpq/be-fsstubs.c b/src/backend/libpq/be-fsstubs.c index 10cc05d5e4b..42da83b9706 100644 --- a/src/backend/libpq/be-fsstubs.c +++ b/src/backend/libpq/be-fsstubs.c @@ -489,7 +489,7 @@ be_lo_export(PG_FUNCTION_ARGS) text *filename = PG_GETARG_TEXT_PP(1); int fd; int nbytes; - char buf[BUFSIZE]; + char *buf; char fnamebuf[MAXPGPATH]; LargeObjectDesc *lobj; mode_t oumask; @@ -525,6 +525,13 @@ be_lo_export(PG_FUNCTION_ARGS) errmsg("could not create server file \"%s\": %m", fnamebuf))); + /* + * The buffer is allocated rather than declared on the stack. A frame + * larger than 4kB that contains a PG_TRY() can be miscompiled by GCC on + * SPARC. + */ + buf = palloc(BUFSIZE); + /* * read in from the inversion file and write to the filesystem */ @@ -540,6 +547,8 @@ be_lo_export(PG_FUNCTION_ARGS) fnamebuf))); } + pfree(buf); + if (CloseTransientFile(fd) != 0) ereport(ERROR, (errcode_for_file_access(),
^ permalink raw reply [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-30 15:59  Greg Burd <greg@burd.me>
  parent: Michael Paquier <michael@paquier.xyz>
  1 sibling, 0 replies; 64+ messages in thread

From: Greg Burd @ 2026-10-30 15:59 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; hlinnaka@iki.fi <hlinnaka@iki.fi>; david@pgbackrest.org <david@pgbackrest.org>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>


On Thursday, October 29th, 2026 at 7:08 PM, Michael Paquier <michael@paquier.xyz> wrote:

> On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:
> > Validation of c97931c16d on the same machine: with it applied, three
> > runs each of both reproductions give the WARNING and ERROR and no
> > crash, and t/010_pg_basebackup.pl passes all 168 tests with no
> > backend crash in the server logs.  With it reverted, the same TAP run
> > logs the SIGSEGV.  The test still reports PASS in that case because
> > it only checks that pg_basebackup fails.
> 
> Ah, thanks for the in-depth investigation.

Hey Michael, sure thing.  It's why I bought the UltraSPARC 45, oddly
enough!

> So the previous guess was really right.

Yes, you have a talent for many things including guessing fixes to
obscure issues.

> Candles work.

Indeed they do. :)

> > I don't see a better fix on our side.  A helper function with
> > the array would also work, but palloc0() is simpler, and both rely on
> > the same thing: no PG_TRY() frame larger than 4kB.  In a core build
> > on this machine, the only other function with a frame that large that
> > calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]).  Its code
> > after sigsetjmp() reloads the constant, so it is fine today, but only
> > by luck of register allocation.  I haven't checked contrib or other
> > GCC versions or as mentioned earlier against their bug system.
> 
> Ugh.  Even if the buffer itself is not used within the try/catch
> block itself?

I could test if you'd like.

> Getting entirely rid of the TRY/CATCH would lead to a
> much nicer result overall: we'd be able to keep the local buffer.  Or
> perhaps go to the simpler solution of a palloc/pfree?  I cannot
> believe that this extra allocation would matter for lo_export() calls.
> 
> If you are proved to be right, it sounds to me that we ought to
> fix and backpatch that as well, especially if there is nothing we can
> do on the Postgres side.

Better safe than sorry.  Even if I submit a bug to GCC and they fix it
there it will take time to get that shipped and you have people who rarely
update, etc. etc. 

> I don't believe in luck and magic when it comes to software programming.

Nope, just 1s and 0s.

> Thoughts?
> --
> Michael

best.

-greg






^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-30 16:04  Greg Burd <greg@burd.me>
  parent: David Steele <david@pgbackrest.org>
  0 siblings, 1 reply; 64+ messages in thread

From: Greg Burd @ 2026-10-30 16:04 UTC (permalink / raw)
  To: David Steele <david@pgbackrest.org>; +Cc: Michael Paquier <michael@paquier.xyz>; Burd, Greg <gregburd@amazon.com>; pgsql-hackers; hlinnaka@iki.fi <hlinnaka@iki.fi>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>


On Friday, October 30th, 2026 at 4:34 AM, David Steele <david@pgbackrest.org> wrote:

> On 10/30/26 00:08, Michael Paquier wrote:
> > On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:
> >> Validation of c97931c16d on the same machine: with it applied, three
> >> runs each of both reproductions give the WARNING and ERROR and no
> >> crash, and t/010_pg_basebackup.pl passes all 168 tests with no
> >> backend crash in the server logs.  With it reverted, the same TAP run
> >> logs the SIGSEGV.  The test still reports PASS in that case because
> >> it only checks that pg_basebackup fails.
> >
> > Ah, thanks for the in-depth investigation.  So the previous guess was
> > really right.  Candles work.
> 
> Thank you, Greg! This is exactly the confirmation we needed.

Hey David, happy to help.  As I said to Michael, now I can write off that
UltraSPARC 45 and the power it gobbles up on my taxes with proof that it
is used for work.

> I would encourage you to file a bug with gcc. I've found a few bugs and
> the gcc folks are really easy to work with. Since you have a repro it
> will be easy to produce the pre-processed output they want.

Sure, I'll carve out some time to do that.

> One bad thing -- even if they fix the bug, distros may choose not to
> back patch it (so it may already be fixed). I had to maintain a patch
> for a gcc bug on ppc64le for years before the affected distros EOL'd.
> That bug limited where I could use const, which was pretty annoying.

But that's the life of an open source developer, right?  I signed up for
it so I can't complain.  Or if I do, you should ignore me.

> >> I don't see a better fix on our side.  A helper function with
> >> the array would also work, but palloc0() is simpler, and both rely on
> >> the same thing: no PG_TRY() frame larger than 4kB.  In a core build
> >> on this machine, the only other function with a frame that large that
> >> calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]).  Its code
> >> after sigsetjmp() reloads the constant, so it is fine today, but only
> >> by luck of register allocation.  I haven't checked contrib or other
> >> GCC versions or as mentioned earlier against their bug system.
> >
> > Ugh.  Even if the buffer itself is not used within the try/catch
> > block itself?
> 
> Apparently so -- the failure of the base backup cancel test showed that.
> The buffer was never used and just the larger frame caused the problem.

I agree.

> > Getting entirely rid of the TRY/CATCH would lead to a
> > much nicer result overall: we'd be able to keep the local buffer.  Or
> > perhaps go to the simpler solution of a palloc/pfree?  I cannot
> > believe that this extra allocation would matter for lo_export() calls.
> 
> palloc/pfree seems like the simpler solution.

+1

> > If you are proved to be right, it sounds to me that we ought to
> > fix and backpatch that as well, especially if there is nothing we can
> > do on the Postgres side.  I don't believe in luck and magic when it
> > comes to software programming.
> I've attached a patch for be_lo_export(). Working around compiler bugs
> is never fun but I'm not sure what choice we have.

I've not reviewed the patch yet, but it is on my queue.

> Regards,
> -David

best.

-greg







^ permalink  raw  reply  [nested|flat] 64+ messages in thread

* Re: Return pg_control from pg_backup_stop().
@ 2026-10-31 00:01  Michael Paquier <michael@paquier.xyz>
  parent: Greg Burd <greg@burd.me>
  0 siblings, 0 replies; 64+ messages in thread

From: Michael Paquier @ 2026-10-31 00:01 UTC (permalink / raw)
  To: Greg Burd <greg@burd.me>; +Cc: David Steele <david@pgbackrest.org>; Burd, Greg <gregburd@amazon.com>; pgsql-hackers; hlinnaka@iki.fi <hlinnaka@iki.fi>; robertmhaas@gmail.com <robertmhaas@gmail.com>; andres@anarazel.de <andres@anarazel.de>; masao.fujii@gmail.com <masao.fujii@gmail.com>; solai.cdac@gmail.com <solai.cdac@gmail.com>

On Fri, Oct 30, 2026 at 04:04:13PM +0000, Greg Burd wrote:
> On Friday, October 30th, 2026 at 4:34 AM, David Steele <david@pgbackrest.org> wrote:
>> I've attached a patch for be_lo_export(). Working around compiler bugs
>> is never fun but I'm not sure what choice we have.
> 
> I've not reviewed the patch yet, but it is on my queue.

Thanks.  Making sure that be_lo_export() can be faulty on SPARC and
also making sure that the palloc/pfree fixes it would be really
useful.  I'm not sure that I'm going to spend cycles spawning an
instance just for this purpose.  That does not feel like the most
useful use of time with everything that is spawning on the lists these
days.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 64+ messages in thread


end of thread, other threads:[~2026-10-31 00:01 UTC | newest]

Thread overview: 64+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-05-17 02:46 Return pg_control from pg_backup_stop(). David Steele <david@pgmasters.net>
2024-10-02 07:15 ` Michael Paquier <michael@paquier.xyz>
2024-10-02 09:03 ` David Steele <david@pgbackrest.org>
2024-10-03 04:45   ` Michael Paquier <michael@paquier.xyz>
2024-10-03 09:11     ` David Steele <david@pgbackrest.org>
2024-11-20 22:44       ` David Steele <david@pgbackrest.org>
2025-01-24 18:43         ` David Steele <david@pgbackrest.org>
2025-08-06 22:30           ` David Steele <david@pgbackrest.org>
2026-02-20 03:10             ` David Steele <david@pgbackrest.org>
2026-02-20 05:47               ` David Steele <david@pgbackrest.org>
2026-03-06 01:27                 ` David Steele <david@pgbackrest.org>
2026-03-17 05:16                   ` Haibo Yan <tristan.yim@gmail.com>
2026-03-17 05:51                     ` Michael Paquier <michael@paquier.xyz>
2026-03-17 07:05                     ` David Steele <david@pgbackrest.org>
2026-03-17 18:50                       ` Haibo Yan <tristan.yim@gmail.com>
2026-03-18 01:43                         ` Michael Paquier <michael@paquier.xyz>
2026-03-18 04:05                           ` David Steele <david@pgbackrest.org>
2026-03-18 04:53                             ` Michael Paquier <michael@paquier.xyz>
2026-03-18 07:35                               ` David Steele <david@pgbackrest.org>
2026-03-18 08:26                                 ` Michael Paquier <michael@paquier.xyz>
2026-03-18 12:26                                   ` David Steele <david@pgbackrest.org>
2026-03-18 21:00                                     ` Corey Huinker <corey.huinker@gmail.com>
2026-03-19 02:55                                       ` David Steele <david@pgbackrest.org>
2026-03-19 04:20                                         ` Corey Huinker <corey.huinker@gmail.com>
2026-04-13 14:55                                     ` David Steele <david@pgbackrest.org>
2026-06-30 05:15                                       ` David Steele <david@pgbackrest.org>
2026-07-14 10:27                                         ` solai v <solai.cdac@gmail.com>
2026-07-20 02:20                                           ` David Steele <david@pgbackrest.org>
2026-10-21 13:07                                           ` David Steele <david@pgbackrest.org>
2026-10-23 12:42                                             ` David Steele <david@pgbackrest.org>
2026-10-23 15:47                                               ` Heikki Linnakangas <hlinnaka@iki.fi>
2026-10-24 05:25                                                 ` Michael Paquier <michael@paquier.xyz>
2026-10-28 00:00                                                   ` Michael Paquier <michael@paquier.xyz>
2026-10-28 06:38                                                     ` Michael Paquier <michael@paquier.xyz>
2026-10-28 08:56                                                       ` David Steele <david@pgbackrest.org>
2026-10-28 10:56                                                         ` Michael Paquier <michael@paquier.xyz>
2026-10-28 15:32                                                           ` David Steele <david@pgbackrest.org>
2026-10-28 16:26                                                             ` Noah Misch <noah@leadboat.com>
2026-10-29 00:48                                                               ` Michael Paquier <michael@paquier.xyz>
2026-10-29 08:27                                                         ` Marcel Hofstetter <hofstetter@jomasoft.ch>
2026-10-29 00:50                                                       ` Michael Paquier <michael@paquier.xyz>
2026-10-29 08:14                                                         ` Michael Paquier <michael@paquier.xyz>
2026-10-24 17:14                                                 ` David Steele <david@pgbackrest.org>
2026-10-23 16:17                                               ` Heikki Linnakangas <hlinnaka@iki.fi>
2026-10-26 11:42                                                 ` David Steele <david@pgbackrest.org>
2026-10-28 09:48                                                   ` Heikki Linnakangas <hlinnaka@iki.fi>
2026-10-28 10:55                                                     ` David Steele <david@pgbackrest.org>
2026-07-28 10:36                                     ` Andy Pogrebnoi <andrew.pogrebnoi@percona.com>
2026-07-30 22:02                                       ` Shaun Thomas <shaun.thomas@pgedge.com>
2026-08-28 02:34                                         ` shihao zhong <zhong950419@gmail.com>
2026-09-15 15:05                                           ` David Steele <david@pgbackrest.org>
2026-09-15 18:13                                             ` Andrey Borodin <x4mmm@yandex-team.ru>
2026-09-15 18:45                                               ` David Steele <david@pgbackrest.org>
2026-09-16 04:14                                                 ` shihao zhong <zhong950419@gmail.com>
2026-09-26 14:58                                                   ` David Steele <david@pgbackrest.org>
2026-09-27 16:02                                                     ` shihao zhong <zhong950419@gmail.com>
2026-09-27 16:13                                                       ` shihao zhong <zhong950419@gmail.com>
2026-09-28 06:03                                                         ` David Steele <david@pgbackrest.org>
2026-10-29 18:11 Re: Return pg_control from pg_backup_stop(). Burd, Greg <gregburd@amazon.com>
2026-10-29 23:08 ` Michael Paquier <michael@paquier.xyz>
2026-10-30 08:33   ` David Steele <david@pgbackrest.org>
2026-10-30 16:04     ` Greg Burd <greg@burd.me>
2026-10-31 00:01       ` Michael Paquier <michael@paquier.xyz>
2026-10-30 15:59   ` Greg Burd <greg@burd.me>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox