public inbox for [email protected]
help / color / mirror / Atom feedFrom: Alexander Korotkov <[email protected]>
To: Tom Lane <[email protected]>
Cc: Alexander Lakhin <[email protected]>
Cc: Vitaly Davydov <[email protected]>
Cc: Amit Kapila <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected]
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
Date: Mon, 16 Jun 2025 04:58:51 +0300
Message-ID: <CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <1d12d2-67235980-35-19a406a0@63439497>
<CAPpHfdtp61pSt72KxxOnnV_Oj4OTAiKYCSTek0Pm87hVDod4Bg@mail.gmail.com>
<CAA4eK1Lmd-stXWrNqX9iXugjY=ahhe7zRfyq+-vN512xvF4G2w@mail.gmail.com>
<CAPpHfdsTiYQPywr4yeqVhUtA+eFdqisqcAOiTb+kiYhi9EVEvw@mail.gmail.com>
<CAPpHfdvk-nfu5xzJyPArxWctofWPR+1L9uJ+yyWFA34n6b5rGQ@mail.gmail.com>
<CAA4eK1Ladnm40UO-YC+c7eBYqjyFQSwg0dn1w_CiaMoBTsJUDA@mail.gmail.com>
<CAPpHfduX--KKX853UwaJ8Cjo5bxbh19V_+McKQ3p9aS6b8T1Wg@mail.gmail.com>
<CAA4eK1+VbBxYY-rG4B7Jq6TsYvdjt3d2nbUrs7p9-1sP5mH+sw@mail.gmail.com>
<CAPpHfdvxqjE+RZdoOODQZAmQip6NpvSH40DuvVFWuf_00ek-yQ@mail.gmail.com>
<1d97ec-6841b600-3-b7b4780@40747246>
<CAA4eK1LvssAcp61qg957xRF0=K4xF0hfJQjQ6c9V7EGHSqraPg@mail.gmail.com>
<28c8bf-68470780-3-51b29480@89454035>
<CAPpHfdt77k3BqD=anh4b7UiRevfPOu2Hcf3rvSZXntxH+4G=Ug@mail.gmail.com>
<[email protected]>
<CAPpHfdvPpg5VUTNOqfKSNx66ij3ZJNZbf4kn6SDcPkdq9FTvng@mail.gmail.com>
<[email protected]>
<[email protected]>
Hi, Tom!
On Sun, Jun 15, 2025 at 7:05 PM Tom Lane <[email protected]> wrote:
> BTW, while you're cleaning up this commit, could you remove the
> excess newlines in some of the "note" commands in 046 and 047, like
>
> note('starting checkpoint\n');
>
> This produces bizarre output, as shown in the buildfarm logs:
Thank you for reporting this. The revised patch is attached. In
addition to reducing tests runtime, it removes excess newlines from
some note() calls. The commit message is here. I'm going to push
this if no objections.
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v2-0001-Improve-runtime-and-output-of-tests-for-replicati.patch (4.9K, ../CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com/2-v2-0001-Improve-runtime-and-output-of-tests-for-replicati.patch)
download | inline diff:
From 5e332ece97a6aefcec569af7e16a4251fc79071c Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 15 Jun 2025 13:54:09 +0300
Subject: [PATCH v2] Improve runtime and output of tests for replication slots
checkpointing.
The TAP tests that verify logical and physical replication slot behavior
during checkpoints (046_checkpoint_logical_slot.pl and
047_checkpoint_physical_slot.pl) inserted two batches of 2 million rows each,
generating approximately 520 MB of WAL. On slow machines, or when compiled
with '-DRELCACHE_FORCE_RELEASE -DCATCACHE_FORCE_RELEASE', this caused the
tests to run for 8-9 minutes and occasionally time out, as seen on the
buildfarm animal prion.
Reduce each INSERT to 50k rows of wider data, which yields approximately
5 segments of WAL. This volume is still sufficient to advance the slot to
the next segment and exercise the code paths under test, but it cuts
the total wall-clock run time.
While here, remove superfluous '\n' characters from several note() calls;
these appeared literally in the build-farm logs and looked odd.
Reported-by: Alexander Lakhin <[email protected]>
Reported-by: Tom Lane <[email protected]>
Discussion: https://postgr.es/m/fbc5d94e-6fbd-4a64-85d4-c9e284a58eb2%40gmail.com
---
src/test/recovery/t/046_checkpoint_logical_slot.pl | 14 +++++++-------
.../recovery/t/047_checkpoint_physical_slot.pl | 8 ++++----
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/src/test/recovery/t/046_checkpoint_logical_slot.pl b/src/test/recovery/t/046_checkpoint_logical_slot.pl
index b4265c4a6a5..964bc34fbe8 100644
--- a/src/test/recovery/t/046_checkpoint_logical_slot.pl
+++ b/src/test/recovery/t/046_checkpoint_logical_slot.pl
@@ -58,23 +58,23 @@ SELECT 1 \watch 0.1
\q
));
-# Insert 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Insert 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Run another checkpoint to set a new restore LSN.
$node->safe_psql('postgres', q{checkpoint});
-# Another 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Another 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Run another checkpoint, this time in the background, and make it wait
# on the injection point) so that the checkpoint stops right before
# removing old WAL segments.
-note('starting checkpoint\n');
+note('starting checkpoint');
my $checkpoint = $node->background_psql('postgres');
$checkpoint->query_safe(
@@ -88,7 +88,7 @@ checkpoint;
));
# Wait until the checkpoint stops right before removing WAL segments.
-note('waiting for injection_point\n');
+note('waiting for injection_point');
$node->wait_for_event('checkpointer', 'checkpoint-before-old-wal-removal');
note('injection_point is reached');
@@ -107,7 +107,7 @@ select count(*) from pg_logical_slot_get_changes('slot_logical', null, null) \wa
));
# Wait until the slot's restart_lsn points to the next WAL segment.
-note('waiting for injection_point\n');
+note('waiting for injection_point');
$node->wait_for_event('client backend',
'logical-replication-slot-advance-segment');
note('injection_point is reached');
diff --git a/src/test/recovery/t/047_checkpoint_physical_slot.pl b/src/test/recovery/t/047_checkpoint_physical_slot.pl
index 454e56b9bd2..3141b256bf3 100644
--- a/src/test/recovery/t/047_checkpoint_physical_slot.pl
+++ b/src/test/recovery/t/047_checkpoint_physical_slot.pl
@@ -43,9 +43,9 @@ $node->safe_psql('postgres',
# Run checkpoint to flush current state to disk and set a baseline.
$node->safe_psql('postgres', q{checkpoint});
-# Insert 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Insert 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,100000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Advance slot to the current position, just to have everything "valid".
@@ -56,9 +56,9 @@ $node->safe_psql('postgres',
# Run another checkpoint to set a new restore LSN.
$node->safe_psql('postgres', q{checkpoint});
-# Another 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Another 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
my $restart_lsn_init = $node->safe_psql('postgres',
--
2.39.5 (Apple Git-154)
view thread (73+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
In-Reply-To: <CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox