Date: Wed, 2 Apr 2025 17:29:00 -0700
From: Noah Misch <noah@leadboat.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Nathan Bossart <nathandbossart@gmail.com>,
	Andres Freund <andres@anarazel.de>,
	Robert Pang <robertpang@google.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	Tom Lane <tgl@sss.pgh.pa.us>, robertmhaas@gmail.com,
	pgsql-hackers@postgresql.org
Subject: Re: Back-patch of: avoid multiple hard links to same WAL file after
 a crash
Message-ID: <20250403002900.5d.nmisch@google.com>
References: <CAJhEC04tBkYPF4q2uS_rCytauvNEVqdBAzasBEokfceFhF=KDQ@mail.gmail.com>
 <Z2L6e1w-xABVTBRR@nathan>
 <7chjz7zeigbsbt7nim4cj6zryflzcpy2lgenas3yh7cpvwf3gb@m6cwqzlvfd6q>
 <Z2Oy1Z2nMVmTM5L5@paquier.xyz>
 <20250306193013.36.nmisch@google.com>
 <Z8n7HopB0bb4_j19@nathan>
 <Z853yPDh1xNWssD_@paquier.xyz>
 <20250311205749.a8.nmisch@google.com>
 <Z9DZY8PVgXI3-b5j@paquier.xyz>
 <20250312012315.fd.nmisch@google.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="402OII5J8w+WWZaJ"
Content-Disposition: inline
In-Reply-To: <20250312012315.fd.nmisch@google.com>
User-Agent: Mutt/2.2.12 (2023-09-09)
Archived-At: <https://www.postgresql.org/message-id/20250403002900.5d.nmisch%40google.com>
Precedence: bulk


--402OII5J8w+WWZaJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Tue, Mar 11, 2025 at 06:23:15PM -0700, Noah Misch wrote:
> On Wed, Mar 12, 2025 at 09:46:27AM +0900, Michael Paquier wrote:
> > On Tue, Mar 11, 2025 at 01:57:49PM -0700, Noah Misch wrote:
> > > Thanks for crafting back-branch versions.  I've queued a task to confirm I get
> > > the same result.
> > 
> > Thanks for that.  That helps a lot.
> 
> I'll let you know when I get there.

Your back-patches are correct.  Thanks.

> > > There's a test case I'll polish, too.
> > 
> > Are you considering the addition of a TAP test in 17~ based on a wait
> > injection point in the checkpointer coupled with a check of the server
> > logs to see if we see the error patterns you've spotted?
> 
> No, nothing involving injection points or otherwise version-specific.

Here it is.  Making it fail three times took looping 1383s, 5841s, and 2594s.
Hence, it couldn't be expected to catch the regression before commit, but it
would have made sufficient buildfarm and CI noise in the day after commit.

--402OII5J8w+WWZaJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
	filename="test-archive-restartpoint-v1.patch"

Author:     Noah Misch <noah@leadboat.com>
Commit:     Noah Misch <noah@leadboat.com>

    Test restartpoints in archive recovery.
    
    v14 commit 1f95181b44c843729caaa688f74babe9403b5850 and its v13 equivalent
    caused timing-dependent failures in archive recovery, at restartpoints.  The
    symptom was "invalid magic number 0000 in log segment X, offset 0",
    "unexpected pageaddr X in log segment Y, offset 0" [X < Y], or an assertion
    failure.  Commit FIXME and predecessors back-patched v15 changes to fix that.
    This test reproduces the problem probabilistically, typically in less than
    1000 iterations of the test.  Hence, buildfarm and CI runs would have surfaced
    enough failures to get attention within a day.  Back-patch to v13 (all
    supported versions).
    
    Reported by Arun Thirupathi.  Reviewed by FIXME.
    
    Discussion: https://postgr.es/m/20250306193013.36.nmisch@google.com

diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 057bcde..cb98376 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -53,6 +53,7 @@ tests += {
       't/042_low_level_backup.pl',
       't/043_no_contrecord_switch.pl',
       't/044_invalidate_inactive_slots.pl',
+      't/045_archive_restartpoint.pl',
     ],
   },
 }
diff --git a/src/test/recovery/t/045_archive_restartpoint.pl b/src/test/recovery/t/045_archive_restartpoint.pl
new file mode 100644
index 0000000..b143bc4
--- /dev/null
+++ b/src/test/recovery/t/045_archive_restartpoint.pl
@@ -0,0 +1,57 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+# Test restartpoints during archive recovery.
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $archive_max_mb = 320;
+my $wal_segsize = 1;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(
+	has_archiving => 1,
+	allows_streaming => 1,
+	extra => [ '--wal-segsize' => $wal_segsize ]);
+$node_primary->start;
+my $backup_name = 'my_backup';
+$node_primary->backup($backup_name);
+
+$node_primary->safe_psql('postgres',
+	('DO $$BEGIN FOR i IN 1..' . $archive_max_mb / $wal_segsize)
+	  . ' LOOP CHECKPOINT; PERFORM pg_switch_wal(); END LOOP; END$$;');
+
+# Force archiving of WAL file containing recovery target
+my $until_lsn = $node_primary->lsn('write');
+$node_primary->safe_psql('postgres', "SELECT pg_switch_wal()");
+$node_primary->stop;
+
+# Archive recovery
+my $node_restore = PostgreSQL::Test::Cluster->new('restore');
+$node_restore->init_from_backup($node_primary, $backup_name,
+	has_restoring => 1);
+$node_restore->append_conf('postgresql.conf',
+	"recovery_target_lsn = '$until_lsn'");
+$node_restore->append_conf('postgresql.conf',
+	'recovery_target_action = pause');
+$node_restore->append_conf('postgresql.conf',
+	'max_wal_size = ' . 2 * $wal_segsize);
+$node_restore->append_conf('postgresql.conf', 'log_checkpoints = on');
+
+$node_restore->start;
+
+# Wait until restore has replayed enough data
+my $caughtup_query =
+  "SELECT '$until_lsn'::pg_lsn <= pg_last_wal_replay_lsn()";
+$node_restore->poll_query_until('postgres', $caughtup_query)
+  or die "Timed out while waiting for restore to catch up";
+
+$node_restore->stop;
+ok(1, 'restore caught up');
+
+done_testing();

--402OII5J8w+WWZaJ--