pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feed019_replslot_limit.pl might fail due to checkpoint skipped
8+ messages / 3 participants
[nested] [flat]
* 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2024-12-17 14:00 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Alexander Lakhin @ 2024-12-17 14:00 UTC (permalink / raw)
To: pgsql-hackers
Hello hackers,
Investigating a recent buildfarm failure [1] with the following
diagnostics:
[12:27:41.437](0.024s) ok 18 - have walreceiver pid 637143
[12:30:42.564](181.127s) not ok 19 - walsender termination logged
[12:30:42.564](0.000s)
[12:30:42.564](0.000s) # Failed test 'walsender termination logged'
# at t/019_replslot_limit.pl line 382.
019_replslot_limit_primary3.log:
2024-12-13 12:27:40.912 ACDT [637093:7] LOG: checkpoint starting: wal
...
2024-12-13 12:27:41.461 ACDT [637182:4] 019_replslot_limit.pl LOG: statement: SELECT pg_logical_emit_message(false, '',
'foo');
2024-12-13 12:27:41.462 ACDT [637182:5] 019_replslot_limit.pl LOG: statement: SELECT pg_switch_wal();
2024-12-13 12:27:41.463 ACDT [637182:6] 019_replslot_limit.pl LOG: disconnection: session time: 0:00:00.003
user=postgres database=postgres host=[local]
2024-12-13 12:27:41.668 ACDT [637093:8] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 1
removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.756 s; sync files=0, longest=0.000 s, average=0.000 s;
distance=1024 kB, estimate=1024 kB; lsn=0/900060, redo lsn=0/700028
2024-12-13 12:27:41.668 ACDT [637093:9] LOG: checkpoints are occurring too frequently (1 second apart)
2024-12-13 12:27:41.668 ACDT [637093:10] HINT: Consider increasing the configuration parameter "max_wal_size".
2024-12-13 12:30:42.565 ACDT [637144:10] standby_3 LOG: terminating walsender process due to replication timeout
### waiting for walsender termination logged timed out after 3 minuts
2024-12-13 12:30:42.565 ACDT [637144:11] standby_3 STATEMENT: START_REPLICATION SLOT "rep3" 0/700000 TIMELINE 1
(note that checkpoint was started at 40.912 and completed at 41.668, i.e.,
it took 0.7 seconds)
I've found that such a failure can occur when the checkpoint, which should
invalidate replication slot (through the call chain: CreateCheckPoint() ->
InvalidateObsoleteReplicationSlots() -> InvalidatePossiblyObsoleteSlot()),
is skipped.
I've managed to reproduce this with the following sleep() added inside
src/backend/access/transam/xlog.c:
@@ -7126,6 +7126,11 @@ CreateCheckPoint(int flags)
* panic. Accordingly, exit critical section while doing it.
*/
END_CRIT_SECTION();
+if (rand() % 2 == 0)
+{
+elog(LOG, "!!!CreateCheckPoint| sleeping...");
+pg_usleep(1000000);
+}
When running just
make -s check -C src/test/recovery/ PROVE_TESTS="t/019*"
in a loop. I got failures on iterations 3, 4, 13, so with debug logging
enabled, I can see the following:
[12:57:41.899](0,009s) ok 18 - have walreceiver pid 509480
[13:00:44.862](182,963s) not ok 19 - walsender termination logged
[13:00:44.862](0,000s)
[13:00:44.862](0,000s) # Failed test 'walsender termination logged'
# at t/019_replslot_limit.pl line 382.
019_replslot_limit_primary3.log:
2024-12-17 12:57:41.695 UTC [][509460:11][checkpointer] LOG: checkpoints are occurring too frequently (0 seconds apart)
2024-12-17 12:57:41.695 UTC [][509460:12][checkpointer] HINT: Consider increasing the configuration parameter
"max_wal_size".
2024-12-17 12:57:41.695 UTC [][509460:13][checkpointer] LOG: checkpoint starting: wal
2024-12-17 12:57:41.695 UTC [][509460:14][checkpointer] LOG: !!!CreateCheckPoint| sleeping...
...
2024-12-17 12:57:41.903 UTC [][509460:15][checkpointer] DEBUG: performing replication slot checkpoint
...
2024-12-17 12:57:41.907 UTC [postgres][509495:5][client backend] [019_replslot_limit.pl] LOG: statement: SELECT
pg_logical_emit_message(false, '', 'foo');
2024-12-17 12:57:41.908 UTC [postgres][509495:6][client backend] [019_replslot_limit.pl] LOG: statement: SELECT
pg_switch_wal();
...
2024-12-17 12:57:41.947 UTC [][509460:16][checkpointer] DEBUG: snapshot of 0+0 running transaction ids (lsn 0/C00060
oldest xid 742 latest complete 741 next xid 742)
2024-12-17 12:57:41.947 UTC [][509460:17][checkpointer] DEBUG: attempting to remove WAL segments older than log file
000000000000000000000009
2024-12-17 12:57:41.947 UTC [][509460:18][checkpointer] DEBUG: removing write-ahead log file "000000010000000000000009"
2024-12-17 12:57:41.948 UTC [][509460:19][checkpointer] DEBUG: SlruScanDirectory invoking callback on pg_subtrans/0000
2024-12-17 12:57:41.948 UTC [][509460:20][checkpointer] LOG: checkpoint complete: wrote 0 buffers (0.0%), wrote 0 SLRU
buffers; 0 WAL file(s) added, 1 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.253 s; sync files=0,
longest=0.000 s, average=0.000 s; distance=1024 kB, estimate=1024 kB; lsn=0/C00060, redo lsn=0/A00028
### the checkpoint, that started before pg_switch_wal(), is completed
2024-12-17 12:57:41.948 UTC [][509460:21][checkpointer] LOG: checkpoints are occurring too frequently (0 seconds apart)
2024-12-17 12:57:41.948 UTC [][509460:22][checkpointer] HINT: Consider increasing the configuration parameter
"max_wal_size".
2024-12-17 12:57:41.948 UTC [][509460:23][checkpointer] DEBUG: checkpoint skipped because system is idle
### the checkpoint, which expected to invalidate the replication slot after pg_switch_wal, is skipped
Reproduced starting from commit 24043c27b (dated 2021-07-23), which added
this test case.
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
Best regards,
Alexander
^ permalink raw reply [nested|flat] 8+ messages in thread
* RE: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-07 06:44 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-08-07 06:44 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: pgsql-hackers
Dear Alexander,
> Investigating a recent buildfarm failure [1] with the following
> diagnostics:
> [12:27:41.437](0.024s) ok 18 - have walreceiver pid 637143
> [12:30:42.564](181.127s) not ok 19 - walsender termination logged
> [12:30:42.564](0.000s)
> [12:30:42.564](0.000s) # Failed test 'walsender termination logged'
> # at t/019_replslot_limit.pl line 382.
Sorry for replying to the old thread, I found it happened again [1].
> I've found that such a failure can occur when the checkpoint, which should
> invalidate replication slot (through the call chain: CreateCheckPoint() ->
> InvalidateObsoleteReplicationSlots() -> InvalidatePossiblyObsoleteSlot()),
> is skipped.
>
> I've managed to reproduce this with the following sleep() added inside
> src/backend/access/transam/xlog.c:
> @@ -7126,6 +7126,11 @@ CreateCheckPoint(int flags)
> * panic. Accordingly, exit critical section while doing it.
> */
> END_CRIT_SECTION();
> +if (rand() % 2 == 0)
> +{
> +elog(LOG, "!!!CreateCheckPoint| sleeping...");
> +pg_usleep(1000000);
> +}
I confirmed that this method is still valid: I used the below variant:
```
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7663,6 +7663,9 @@ CreateCheckPoint(int flags)
*/
END_CRIT_SECTION();
+ elog(LOG, "checkpoint paused");
+ pg_usleep(5 * 1000000L);
```
IIUC, the workaround is to force the CHECKPOINT after advancing WAL segments.
Also, we can increase the max_wal_size to avoid the checkpoint due to WAL files.
Please see attached. How do you feel?
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] 0001-Stablize-019_replslot_limit.patch (2.0K, ../../OS9PR01MB121491143DB264B497B55779BF5D12@OS9PR01MB12149.jpnprd01.prod.outlook.com/2-0001-Stablize-019_replslot_limit.patch)
download | inline diff:
From ebbc32bd3af9fe281c28f83f30ffaa3bd6378254 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH] Stablize 019_replslot_limit
---
src/test/recovery/t/019_replslot_limit.pl | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index a412faf51c6..5772073f016 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -302,12 +302,14 @@ if ($PostgreSQL::Test::Utils::windows_os)
# Get a slot terminated while the walsender is active
# We do this by sending SIGSTOP to the walsender. Skip this on Windows.
+# max_wal_size is set to the larger value to prevent other checkpoints from
+# occurring
my $node_primary3 = PostgreSQL::Test::Cluster->new('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
min_wal_size = 2MB
- max_wal_size = 2MB
+ max_wal_size = 40MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -374,6 +376,15 @@ $logstart = -s $node_primary3->logfile;
kill 'STOP', $senderpid, $receiverpid;
$node_primary3->advance_wal(2);
+# Start a psql session to run the checkpoint in the background. It cannot
+# complete while the walsender is stopped.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $msg_logged = 0;
my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
@@ -397,6 +408,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$msg_logged = 0;
$max_attempts = $PostgreSQL::Test::Utils::timeout_default;
--
2.52.0
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-09 10:00 Alexander Lakhin <exclusion@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 8+ messages in thread
From: Alexander Lakhin @ 2026-08-09 10:00 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: pgsql-hackers
Dear Kuroda-san,
07.08.2026 09:44, Hayato Kuroda (Fujitsu) wrote:
> IIUC, the workaround is to force the CHECKPOINT after advancing WAL segments.
> Also, we can increase the max_wal_size to avoid the checkpoint due to WAL files.
>
> Please see attached. How do you feel?
>
> [1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Thank you for paying attention to the issue!
I've tested $checkpoint->query_until() and confirmed that this approach
(which is also used in 4 other tests) eliminates the failure. I haven't
observed an effect of "max_wal_size = 40MB", so I'm not sure about it.
Do you have a scenario in mind, when it could be useful for the test?
Best regards,
Alexander
^ permalink raw reply [nested|flat] 8+ messages in thread
* RE: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-12 03:06 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-08-12 03:06 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: pgsql-hackers
Dear Alexander,
> I've tested $checkpoint->query_until() and confirmed that this approach
> (which is also used in 4 other tests) eliminates the failure.
Thanks!
> I haven't
> observed an effect of "max_wal_size = 40MB", so I'm not sure about it.
> Do you have a scenario in mind, when it could be useful for the test?
Actually the test can pass without increasing the max_wal_size. Ultimately, we do
not have to set both min_wal_size and max_wal_size: we enforce the CHECKPOINT
here. My fresh brain feels that such unnecessary settings can be avoided, done
in v2.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v2-0001-Stablize-019_replslot_limit.patch (1.7K, ../../OS9PR01MB12149B9B394A35D73E4D635F6F5DC2@OS9PR01MB12149.jpnprd01.prod.outlook.com/2-v2-0001-Stablize-019_replslot_limit.patch)
download | inline diff:
From ed93abb97f7ce745bbe89b2cc7cce2139e504757 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v2] Stablize 019_replslot_limit
---
src/test/recovery/t/019_replslot_limit.pl | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index a412faf51c6..1fe698fcc04 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -306,8 +306,6 @@ my $node_primary3 = PostgreSQL::Test::Cluster->new('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -374,6 +372,15 @@ $logstart = -s $node_primary3->logfile;
kill 'STOP', $senderpid, $receiverpid;
$node_primary3->advance_wal(2);
+# Start a psql session to run the checkpoint in the background. It cannot
+# complete while the walsender is stopped.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $msg_logged = 0;
my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
@@ -397,6 +404,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$msg_logged = 0;
$max_attempts = $PostgreSQL::Test::Utils::timeout_default;
--
2.52.0
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-12 08:01 Fujii Masao <masao.fujii@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 8+ messages in thread
From: Fujii Masao @ 2026-08-12 08:01 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers
On Wed, Aug 12, 2026 at 12:06 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
> Actually the test can pass without increasing the max_wal_size. Ultimately, we do
> not have to set both min_wal_size and max_wal_size: we enforce the CHECKPOINT
> here. My fresh brain feels that such unnecessary settings can be avoided, done
> in v2.
Thanks for updating the patch! The test changes look good to me.
I also updated the commit message and adjusted the comment for the new
test. Attached is v3.
Barring any objections, I'm thinking to commit it.
Regards,
--
Fujii Masao
From b82a2656d4009fc986f01d6f353bdef5751d54eb Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v3] Stabilize 019_replslot_limit
The test assumed that advancing WAL would lead to a checkpoint that
invalidates the obsolete replication slot. If a checkpoint that started
before the WAL switch completes first, the following checkpoint can be
skipped as idle, so the expected walsender termination is not logged.
Force a CHECKPOINT in a background psql session after advancing WAL, so
the slot invalidation is exercised deterministically.
This has been observed on buildfarm members alligator and partridge:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/0b07ead5-a5da-445e-9698-a7d340708bdf@gmail.com
Backpatch-through: 14
---
src/test/recovery/t/019_replslot_limit.pl | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 1eec98f2c9e..e78b548b213 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -314,8 +314,6 @@ my $node_primary3 = get_new_node('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -345,6 +343,16 @@ $logstart = get_log_size($node_primary3);
kill 'STOP', $senderpid, $receiverpid;
advance_wal($node_primary3, 2);
+# Run CHECKPOINT in the background. It is expected to reach slot
+# invalidation, signal the stopped walsender, and then wait until the
+# walsender releases the slot.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $max_attempts = $TestLib::timeout_default;
while ($max_attempts-- >= 0)
{
@@ -366,6 +374,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$max_attempts = $TestLib::timeout_default;
while ($max_attempts-- >= 0)
--
2.55.0
From 1b2b08dd2599ec187a03f695ae45d6290eeaf9c6 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v3] Stabilize 019_replslot_limit
The test assumed that advancing WAL would lead to a checkpoint that
invalidates the obsolete replication slot. If a checkpoint that started
before the WAL switch completes first, the following checkpoint can be
skipped as idle, so the expected walsender termination is not logged.
Force a CHECKPOINT in a background psql session after advancing WAL, so
the slot invalidation is exercised deterministically.
This has been observed on buildfarm members alligator and partridge:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/0b07ead5-a5da-445e-9698-a7d340708bdf@gmail.com
Backpatch-through: 14
---
src/test/recovery/t/019_replslot_limit.pl | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 4ec1a9ab33a..ee5a08f57c0 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -309,8 +309,6 @@ my $node_primary3 = PostgreSQL::Test::Cluster->new('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -378,6 +376,16 @@ $logstart = get_log_size($node_primary3);
kill 'STOP', $senderpid, $receiverpid;
advance_wal($node_primary3, 2);
+# Run CHECKPOINT in the background. It is expected to reach slot
+# invalidation, signal the stopped walsender, and then wait until the
+# walsender releases the slot.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
{
@@ -399,6 +407,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
--
2.55.0
Attachments:
[text/plain] v3-REL14-0001-Stabilize-019_replslot_limit.txt (2.7K, ../../CAHGQGwF_O7ZxAdW-TRQ1K83nm9OcHyAhAaVB4bXu_j+GaskvQQ@mail.gmail.com/2-v3-REL14-0001-Stabilize-019_replslot_limit.txt)
download | inline diff:
From b82a2656d4009fc986f01d6f353bdef5751d54eb Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v3] Stabilize 019_replslot_limit
The test assumed that advancing WAL would lead to a checkpoint that
invalidates the obsolete replication slot. If a checkpoint that started
before the WAL switch completes first, the following checkpoint can be
skipped as idle, so the expected walsender termination is not logged.
Force a CHECKPOINT in a background psql session after advancing WAL, so
the slot invalidation is exercised deterministically.
This has been observed on buildfarm members alligator and partridge:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/0b07ead5-a5da-445e-9698-a7d340708bdf@gmail.com
Backpatch-through: 14
---
src/test/recovery/t/019_replslot_limit.pl | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 1eec98f2c9e..e78b548b213 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -314,8 +314,6 @@ my $node_primary3 = get_new_node('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -345,6 +343,16 @@ $logstart = get_log_size($node_primary3);
kill 'STOP', $senderpid, $receiverpid;
advance_wal($node_primary3, 2);
+# Run CHECKPOINT in the background. It is expected to reach slot
+# invalidation, signal the stopped walsender, and then wait until the
+# walsender releases the slot.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $max_attempts = $TestLib::timeout_default;
while ($max_attempts-- >= 0)
{
@@ -366,6 +374,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$max_attempts = $TestLib::timeout_default;
while ($max_attempts-- >= 0)
--
2.55.0
[application/octet-stream] v3-0001-Stabilize-019_replslot_limit.patch (2.7K, ../../CAHGQGwF_O7ZxAdW-TRQ1K83nm9OcHyAhAaVB4bXu_j+GaskvQQ@mail.gmail.com/3-v3-0001-Stabilize-019_replslot_limit.patch)
download | inline diff:
From 663979a68cc03a8ab99716c194e0d032da06c5bf Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v3] Stabilize 019_replslot_limit
The test assumed that advancing WAL would lead to a checkpoint that
invalidates the obsolete replication slot. If a checkpoint that started
before the WAL switch completes first, the following checkpoint can be
skipped as idle, so the expected walsender termination is not logged.
Force a CHECKPOINT in a background psql session after advancing WAL, so
the slot invalidation is exercised deterministically.
This has been observed on buildfarm members alligator and partridge:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/0b07ead5-a5da-445e-9698-a7d340708bdf@gmail.com
Backpatch-through: 14
---
src/test/recovery/t/019_replslot_limit.pl | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index a412faf51c6..aa4217864a1 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -306,8 +306,6 @@ my $node_primary3 = PostgreSQL::Test::Cluster->new('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -374,6 +372,16 @@ $logstart = -s $node_primary3->logfile;
kill 'STOP', $senderpid, $receiverpid;
$node_primary3->advance_wal(2);
+# Run CHECKPOINT in the background. It is expected to reach slot
+# invalidation, signal the stopped walsender, and then wait until the
+# walsender releases the slot.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $msg_logged = 0;
my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
@@ -397,6 +405,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$msg_logged = 0;
$max_attempts = $PostgreSQL::Test::Utils::timeout_default;
--
2.55.0
[text/plain] v3-REL15-0001-Stabilize-019_replslot_limit.txt (2.7K, ../../CAHGQGwF_O7ZxAdW-TRQ1K83nm9OcHyAhAaVB4bXu_j+GaskvQQ@mail.gmail.com/4-v3-REL15-0001-Stabilize-019_replslot_limit.txt)
download | inline diff:
From 1b2b08dd2599ec187a03f695ae45d6290eeaf9c6 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 7 Aug 2026 15:35:31 +0900
Subject: [PATCH v3] Stabilize 019_replslot_limit
The test assumed that advancing WAL would lead to a checkpoint that
invalidates the obsolete replication slot. If a checkpoint that started
before the WAL switch completes first, the following checkpoint can be
skipped as idle, so the expected walsender termination is not logged.
Force a CHECKPOINT in a background psql session after advancing WAL, so
the slot invalidation is exercised deterministically.
This has been observed on buildfarm members alligator and partridge:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=alligator&dt=2024-12-13%2001%3A24%3A58
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=partridge&dt=2026-08-06%2018%3A00%3A11
Backpatch to all supported versions.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@gmail.com>
Discussion: https://postgr.es/m/0b07ead5-a5da-445e-9698-a7d340708bdf@gmail.com
Backpatch-through: 14
---
src/test/recovery/t/019_replslot_limit.pl | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 4ec1a9ab33a..ee5a08f57c0 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -309,8 +309,6 @@ my $node_primary3 = PostgreSQL::Test::Cluster->new('primary3');
$node_primary3->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
$node_primary3->append_conf(
'postgresql.conf', qq(
- min_wal_size = 2MB
- max_wal_size = 2MB
log_checkpoints = yes
max_slot_wal_keep_size = 1MB
));
@@ -378,6 +376,16 @@ $logstart = get_log_size($node_primary3);
kill 'STOP', $senderpid, $receiverpid;
advance_wal($node_primary3, 2);
+# Run CHECKPOINT in the background. It is expected to reach slot
+# invalidation, signal the stopped walsender, and then wait until the
+# walsender releases the slot.
+my $checkpoint = $node_primary3->background_psql('postgres');
+$checkpoint->query_until(
+ qr/starting_checkpoint/, q(
+ \echo starting_checkpoint
+ CHECKPOINT;
+));
+
my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
{
@@ -399,6 +407,7 @@ $node_primary3->poll_query_until('postgres',
"SELECT wal_status FROM pg_replication_slots WHERE slot_name = 'rep3'",
"lost")
or die "timed out waiting for slot to be lost";
+$checkpoint->quit;
$max_attempts = $PostgreSQL::Test::Utils::timeout_default;
while ($max_attempts-- >= 0)
--
2.55.0
^ permalink raw reply [nested|flat] 8+ messages in thread
* RE: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-12 10:31 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-08-12 10:31 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers
Dear Fujii-san,
> Thanks for updating the patch! The test changes look good to me.
>
> I also updated the commit message and adjusted the comment for the new
> test. Attached is v3.
Thanks, LGTM. One note: I tried to reproduce the test failure for all supported
branches by [1], but it could not on PG17 and prior ones. Bute ideally it could happen
for all of them.
[1]: https://www.postgresql.org/message-id/OS9PR01MB121491143DB264B497B55779BF5D12%40OS9PR01MB12149.jpnpr...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-27 15:01 Fujii Masao <masao.fujii@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 8+ messages in thread
From: Fujii Masao @ 2026-08-27 15:01 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers
On Wed, Aug 12, 2026 at 7:31 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
> Thanks, LGTM. One note: I tried to reproduce the test failure for all supported
> branches by [1], but it could not on PG17 and prior ones. Bute ideally it could happen
> for all of them.
I've pushed the patch and backpatch it to v14. Thanks!
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 8+ messages in thread
* RE: 019_replslot_limit.pl might fail due to checkpoint skipped
@ 2026-08-28 09:04 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Fujii Masao <masao.fujii@gmail.com>
0 siblings, 0 replies; 8+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-08-28 09:04 UTC (permalink / raw)
To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers
Dear Fujii-san,
> I've pushed the patch and backpatch it to v14. Thanks!
Thanks for pushing. I checked BF and no similar reported for now.
Let's keep in watch.
BTW, I noticed that the wikipage [1] has already been updated.
Not sure how did it, but thanks for the quick work.
[1]: https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-08-28 09:04 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-12-17 14:00 019_replslot_limit.pl might fail due to checkpoint skipped Alexander Lakhin <exclusion@gmail.com>
2026-08-07 06:44 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2026-08-09 10:00 ` Alexander Lakhin <exclusion@gmail.com>
2026-08-12 03:06 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2026-08-12 08:01 ` Fujii Masao <masao.fujii@gmail.com>
2026-08-12 10:31 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2026-08-27 15:01 ` Fujii Masao <masao.fujii@gmail.com>
2026-08-28 09:04 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox