public inbox for [email protected]
help / color / mirror / Atom feedFrom: Alexander Korotkov <[email protected]>
To: Hayato Kuroda (Fujitsu) <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Alexander Lakhin <[email protected]>
Cc: Vitaly Davydov <[email protected]>
Cc: Amit Kapila <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
Date: Mon, 16 Jun 2025 14:16:42 +0300
Message-ID: <CAPpHfdsdJ4YPTzH=cnany5_LjCNb_Z9e+Oucv53qYQHaHzJhag@mail.gmail.com> (raw)
In-Reply-To: <OSCPR01MB1496667BEA93302F8903EE512F570A@OSCPR01MB14966.jpnprd01.prod.outlook.com>
References: <1d12d2-67235980-35-19a406a0@63439497>
<CAPpHfdtp61pSt72KxxOnnV_Oj4OTAiKYCSTek0Pm87hVDod4Bg@mail.gmail.com>
<CAA4eK1Lmd-stXWrNqX9iXugjY=ahhe7zRfyq+-vN512xvF4G2w@mail.gmail.com>
<CAPpHfdsTiYQPywr4yeqVhUtA+eFdqisqcAOiTb+kiYhi9EVEvw@mail.gmail.com>
<CAPpHfdvk-nfu5xzJyPArxWctofWPR+1L9uJ+yyWFA34n6b5rGQ@mail.gmail.com>
<CAA4eK1Ladnm40UO-YC+c7eBYqjyFQSwg0dn1w_CiaMoBTsJUDA@mail.gmail.com>
<CAPpHfduX--KKX853UwaJ8Cjo5bxbh19V_+McKQ3p9aS6b8T1Wg@mail.gmail.com>
<CAA4eK1+VbBxYY-rG4B7Jq6TsYvdjt3d2nbUrs7p9-1sP5mH+sw@mail.gmail.com>
<CAPpHfdvxqjE+RZdoOODQZAmQip6NpvSH40DuvVFWuf_00ek-yQ@mail.gmail.com>
<1d97ec-6841b600-3-b7b4780@40747246>
<CAA4eK1LvssAcp61qg957xRF0=K4xF0hfJQjQ6c9V7EGHSqraPg@mail.gmail.com>
<28c8bf-68470780-3-51b29480@89454035>
<CAPpHfdt77k3BqD=anh4b7UiRevfPOu2Hcf3rvSZXntxH+4G=Ug@mail.gmail.com>
<[email protected]>
<CAPpHfdvPpg5VUTNOqfKSNx66ij3ZJNZbf4kn6SDcPkdq9FTvng@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAPpHfdvJMe4i0_wp1tq2ZGSDKqUHKPwkfD_Y6KyJ3XQRFvc0cQ@mail.gmail.com>
<OSCPR01MB1496667BEA93302F8903EE512F570A@OSCPR01MB14966.jpnprd01.prod.outlook.com>
Dear Kuroda-san,
On Mon, Jun 16, 2025 at 12:11 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
> Thanks for pushing the fix patch! BTW, I have few comments for your commits.
> Can you check and include them if needed?
>
> 01.
> ```
> $node->append_conf('postgresql.conf',
> "shared_preload_libraries = 'injection_points'");
> ```
>
> No need to set shared_preload_libraries in 046/047. ISTM it must be set when we
> enable the statistics.
>
> 02.
> We should also check whether the injection_points can be installed or not.
> You can check check_extension() and callers.
Thank you! All of these totally make sense. The updated patch is attached.
------
Regards,
Alexander Korotkov
Supabase
Attachments:
[application/octet-stream] v3-0001-Improve-runtime-and-output-of-tests-for-replicati.patch (6.4K, ../CAPpHfdsdJ4YPTzH=cnany5_LjCNb_Z9e+Oucv53qYQHaHzJhag@mail.gmail.com/2-v3-0001-Improve-runtime-and-output-of-tests-for-replicati.patch)
download | inline diff:
From 7b02905e9bad091a269cae3194f4715ed41dffe0 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 15 Jun 2025 13:54:09 +0300
Subject: [PATCH v3] Improve runtime and output of tests for replication slots
checkpointing.
The TAP tests that verify logical and physical replication slot behavior
during checkpoints (046_checkpoint_logical_slot.pl and
047_checkpoint_physical_slot.pl) inserted two batches of 2 million rows each,
generating approximately 520 MB of WAL. On slow machines, or when compiled
with '-DRELCACHE_FORCE_RELEASE -DCATCACHE_FORCE_RELEASE', this caused the
tests to run for 8-9 minutes and occasionally time out, as seen on the
buildfarm animal prion.
Reduce each INSERT to 50k rows of wider data, which yields approximately
5 segments of WAL. This volume is still sufficient to advance the slot to
the next segment and exercise the code paths under test, but it cuts
the total wall-clock run time.
While here, remove superfluous '\n' characters from several note() calls;
these appeared literally in the build-farm logs and looked odd. Also, remove
excessive 'shared_preload_libraries' GUC from the config and add a check for
'injection_points' extension availability.
Reported-by: Alexander Lakhin <[email protected]>
Reported-by: Tom Lane <[email protected]>
Discussion: https://postgr.es/m/fbc5d94e-6fbd-4a64-85d4-c9e284a58eb2%40gmail.com
---
.../recovery/t/046_checkpoint_logical_slot.pl | 25 ++++++++++++-------
.../t/047_checkpoint_physical_slot.pl | 19 +++++++++-----
2 files changed, 29 insertions(+), 15 deletions(-)
diff --git a/src/test/recovery/t/046_checkpoint_logical_slot.pl b/src/test/recovery/t/046_checkpoint_logical_slot.pl
index b4265c4a6a5..86cdc1a2d2c 100644
--- a/src/test/recovery/t/046_checkpoint_logical_slot.pl
+++ b/src/test/recovery/t/046_checkpoint_logical_slot.pl
@@ -21,10 +21,17 @@ my ($node, $result);
$node = PostgreSQL::Test::Cluster->new('mike');
$node->init;
-$node->append_conf('postgresql.conf',
- "shared_preload_libraries = 'injection_points'");
$node->append_conf('postgresql.conf', "wal_level = 'logical'");
$node->start;
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
$node->safe_psql('postgres', q(CREATE EXTENSION injection_points));
# Create a simple table to generate data into.
@@ -58,23 +65,23 @@ SELECT 1 \watch 0.1
\q
));
-# Insert 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Insert 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Run another checkpoint to set a new restore LSN.
$node->safe_psql('postgres', q{checkpoint});
-# Another 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Another 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Run another checkpoint, this time in the background, and make it wait
# on the injection point) so that the checkpoint stops right before
# removing old WAL segments.
-note('starting checkpoint\n');
+note('starting checkpoint');
my $checkpoint = $node->background_psql('postgres');
$checkpoint->query_safe(
@@ -88,7 +95,7 @@ checkpoint;
));
# Wait until the checkpoint stops right before removing WAL segments.
-note('waiting for injection_point\n');
+note('waiting for injection_point');
$node->wait_for_event('checkpointer', 'checkpoint-before-old-wal-removal');
note('injection_point is reached');
@@ -107,7 +114,7 @@ select count(*) from pg_logical_slot_get_changes('slot_logical', null, null) \wa
));
# Wait until the slot's restart_lsn points to the next WAL segment.
-note('waiting for injection_point\n');
+note('waiting for injection_point');
$node->wait_for_event('client backend',
'logical-replication-slot-advance-segment');
note('injection_point is reached');
diff --git a/src/test/recovery/t/047_checkpoint_physical_slot.pl b/src/test/recovery/t/047_checkpoint_physical_slot.pl
index 454e56b9bd2..5ff7ecdb905 100644
--- a/src/test/recovery/t/047_checkpoint_physical_slot.pl
+++ b/src/test/recovery/t/047_checkpoint_physical_slot.pl
@@ -21,10 +21,17 @@ my ($node, $result);
$node = PostgreSQL::Test::Cluster->new('mike');
$node->init;
-$node->append_conf('postgresql.conf',
- "shared_preload_libraries = 'injection_points'");
$node->append_conf('postgresql.conf', "wal_level = 'replica'");
$node->start;
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
$node->safe_psql('postgres', q(CREATE EXTENSION injection_points));
# Create a simple table to generate data into.
@@ -43,9 +50,9 @@ $node->safe_psql('postgres',
# Run checkpoint to flush current state to disk and set a baseline.
$node->safe_psql('postgres', q{checkpoint});
-# Insert 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Insert 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,100000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
# Advance slot to the current position, just to have everything "valid".
@@ -56,9 +63,9 @@ $node->safe_psql('postgres',
# Run another checkpoint to set a new restore LSN.
$node->safe_psql('postgres', q{checkpoint});
-# Another 2M rows; that's about 260MB (~20 segments) worth of WAL.
+# Another 50K rows; that's about 86MB (~5 segments) worth of WAL.
$node->safe_psql('postgres',
- q{insert into t (b) select md5(i::text) from generate_series(1,1000000) s(i)}
+ q{insert into t (b) select repeat(md5(i::text),50) from generate_series(1,50000) s(i)}
);
my $restart_lsn_init = $node->safe_psql('postgres',
--
2.39.5 (Apple Git-154)
view thread (73+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly
In-Reply-To: <CAPpHfdsdJ4YPTzH=cnany5_LjCNb_Z9e+Oucv53qYQHaHzJhag@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox