pg.ddx.io pgsql-bugs@postgresql.org mailing list archive help / color / mirror / Atom feed From: Alexander Lakhin <exclusion@gmail.com>
To: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18426: Canceling vacuum while truncating a relation leads to standby PANIC
Date: Wed, 10 Apr 2024 21:00:00 +0300
Message-ID: <b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com> (raw )
In-Reply-To: <18426-2d18da6586f152d6@postgresql.org >
References: <18426-2d18da6586f152d6@postgresql.org >
10.04.2024 14:00, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 18426
> ...
> A demo test for the issue to follow...
Please try the attached patch (for REL_16_STABLE), which:
1) adds a delay inside InvalidateBuffer() to allow for canceling the
operation here.
2) invokes ProcessInterrupts() which would be called if
(buf_state & BM_IO_IN_PROGRESS) inside ConditionVariableSleep() ->
ConditionVariableTimedSleep().
(If you find these changes non-legitimate, I'll look for another approach.)
3) contains test 099_stream_vacuum, based on 027_stream_regress.
make -s check -C src/test/recovery/ PROVE_TESTS="t/099*"
fails for me (roughly on 1 of 2 runs) just like in the case observed in
the wild:
...
2024-04-10 17:20:54.521 UTC|||6616ca5a.26a393|WARNING: page 90 of relation base/16402/16929 does not exist
2024-04-10 17:20:54.521 UTC|||6616ca5a.26a393|CONTEXT: WAL redo at 0/DCD12AC0 for Heap2/VISIBLE:
snapshotConflictHorizon: 0, flags: 0x03; blkref #0: rel 1663/16402/16929, fork 2, blk 0; blkref #1: rel
1663/16402/16929, blk 90
2024-04-10 17:20:54.521 UTC|||6616ca5a.26a393|PANIC: WAL contains references to invalid pages
2024-04-10 17:20:54.521 UTC|||6616ca5a.26a393|CONTEXT: WAL redo at 0/DCD12AC0 for Heap2/VISIBLE:
snapshotConflictHorizon: 0, flags: 0x03; blkref #0: rel 1663/16402/16929, fork 2, blk 0; blkref #1: rel
1663/16402/16929, blk 90
2024-04-10 17:20:54.532 UTC|||6616ca5a.26a390|LOG: startup process (PID 2532243) was terminated by signal 6: Aborted
...
Best regards,
Alexander
Attachments:
[text/x-patch] bug-18426-demo.patch (3.2K, ../b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com/2-bug-18426-demo.patch)
download | inline diff:
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e066a3f888..60c3d11ec5 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1469,6 +1469,7 @@ retry:
*/
if (BUF_STATE_GET_REFCOUNT(buf_state) != 0)
{
+pg_usleep(1000000L);
UnlockBufHdr(buf, buf_state);
LWLockRelease(oldPartitionLock);
/* safety check: should definitely not be our *own* pin */
@@ -5080,6 +5081,7 @@ WaitIO(BufferDesc *buf)
buf_state = LockBufHdr(buf);
UnlockBufHdr(buf, buf_state);
+ProcessInterrupts();
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
diff --git a/src/test/recovery/t/099_stream_vacuum.pl b/src/test/recovery/t/099_stream_vacuum.pl
new file mode 100644
index 0000000000..a0ed58a3a2
--- /dev/null
+++ b/src/test/recovery/t/099_stream_vacuum.pl
@@ -0,0 +1,58 @@
+# Run the standard regression tests with streaming replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use File::Basename;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+$node_primary->append_conf('postgresql.conf', qq{
+fsync = off
+
+max_connections = 200
+
+log_min_messages = INFO
+log_connections = on
+log_disconnections = on
+log_line_prefix = '%m|%u|%d|%c|'
+log_statement = 'all'
+
+shared_buffers = 1MB
+});
+
+# Increase some settings that Cluster->new makes too low by default.
+$node_primary->adjust_conf('postgresql.conf', 'max_connections', '25');
+
+$node_primary->start;
+
+is( $node_primary->psql(
+ 'postgres',
+ qq[SELECT pg_create_physical_replication_slot('standby');]),
+ 0,
+ 'physical slot created on primary');
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby = PostgreSQL::Test::Cluster->new('standby');
+$node_standby->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby->append_conf('postgresql.conf', "primary_slot_name = standby");
+$node_standby->append_conf('postgresql.conf', "max_standby_streaming_delay = 600s");
+$node_standby->start;
+
+my $setenv = "PGHOST=" . $node_primary->host . " PGPORT=" . $node_primary->port;
+system(qq(sh -c "$setenv ./vacuum-test.sh"));
+
+# Wait for standby to catch up
+$node_primary->wait_for_replay_catchup($node_standby);
+
+$node_standby->stop;
+$node_primary->stop;
+
+done_testing();
diff --git a/src/test/recovery/vacuum-test.sh b/src/test/recovery/vacuum-test.sh
new file mode 100755
index 0000000000..34061a6745
--- /dev/null
+++ b/src/test/recovery/vacuum-test.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+numclients=20
+for ((c=1; c<=numclients;c++)); do createdb db$c; done
+
+for ((i=1;i<=50;i++)); do
+ echo "I $i"
+ for ((c=1; c<=numclients;c++)); do
+ echo "
+DROP TABLE IF EXISTS t;
+
+CREATE TABLE t(i int);
+INSERT INTO t SELECT g FROM generate_series(1, 100000) g;
+
+DELETE FROM t WHERE i > 1000;
+SET statement_timeout = '0.5s';
+VACUUM (VERBOSE, ANALYZE) t;
+RESET statement_timeout;
+VACUUM (VERBOSE) t;
+" | psql db$c >/dev/null &
+ done
+ wait
+done
view thread (6+ messages) latest in thread
Message-ID: <b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com>
Permalink: ../b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com/
Also on: postgresql.org/message-id/b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: exclusion@gmail.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18426: Canceling vacuum while truncating a relation leads to standby PANIC
In-Reply-To: <b57aa365-83b6-5a5b-fb75-be5862a04600@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox