pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
25+ messages / 7 participants
[nested] [flat]
* BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-08 20:41 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 2 replies; 25+ messages in thread
From: PG Bug reporting form @ 2022-02-08 20:41 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: bench@silentmedia.com
The following bug has been logged on the website:
Bug reference: 17401
Logged by: Ben Chobot
Email address: bench@silentmedia.com
PostgreSQL version: 12.9
Operating system: Linux (Ubuntu)
Description:
This bug is is almost identical to BUG #17389, which I filed blaming
pg_repack; however, further testing shows the same symptoms using vanilla
REINDEX TABLE CONCURRENTLY.
1. Put some data in a table with a single btree index:
create table public.simple_test (id int primary key);
insert into public.simple_test(id) (select generate_series(1,1000));
2. Set up streaming replication to a secondary db.
3. In a loop on the primary, concurrently REINDEX that table:
while `true`; do psql -tAc "select now(),relfilenode from pg_class where
relname='simple_test_pkey'" >> log; psql -tAc "reindex table concurrently
public.simple_test"; done
4. In a loop on the secondary, have psql query the secondary db for an
indexed value of that table:
while `true`; do psql -tAc "select count(*) from simple_test where id='3';
select relfilenode from pg_class where relname='simple_test_pkey'" || break;
done; date
With those 4 steps, the client on the replica will reliably fail to open the
OID of the index within 30 minutes of looping. ("ERROR: could not open
relation with OID 6715827") When we run the same client loop on the primary
instead of the replica, or if we reindex without the CONCURRENTLY clause,
then the client loop will run for hours without fail, but neither of those
workarounds are options for us in production.
Like I said before, this isn't a new problem - we've seen it since at least
9.5 - but pre-12 we saw it using pg_repack, which is an easy (and
reasonable) scapegoat. But now that we've upgraded to 12 and are still
seeing it using vanilla concurrent reindexing, it seems more clear this is
an actual postgres bug?
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-08 21:23 Peter Geoghegan <pg@bowt.ie>
parent: PG Bug reporting form <noreply@postgresql.org>
1 sibling, 1 reply; 25+ messages in thread
From: Peter Geoghegan @ 2022-02-08 21:23 UTC (permalink / raw)
To: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; +Cc: Michael Paquier <michael@paquier.xyz>
On Tue, Feb 8, 2022 at 1:01 PM PG Bug reporting form
<noreply@postgresql.org> wrote:
> With those 4 steps, the client on the replica will reliably fail to open the
> OID of the index within 30 minutes of looping. ("ERROR: could not open
> relation with OID 6715827") When we run the same client loop on the primary
> instead of the replica, or if we reindex without the CONCURRENTLY clause,
> then the client loop will run for hours without fail, but neither of those
> workarounds are options for us in production.
I find the idea that we'd fail to WAL-log information that is needed
during Hot Standby (to prevent this race condition) plausible.
Michael?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 01:35 Michael Paquier <michael@paquier.xyz>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 25+ messages in thread
From: Michael Paquier @ 2022-02-09 01:35 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Feb 08, 2022 at 01:23:34PM -0800, Peter Geoghegan wrote:
> I find the idea that we'd fail to WAL-log information that is needed
> during Hot Standby (to prevent this race condition) plausible.
> Michael?
Yeah, REINDEX relies on some existing index definition, so it feels
like we are missing a piece related to invalid indexes in all that. A
main difference is the lock level, as exclusive locks are getting
logged so the standby can react and wait on that. The 30-minute mark
is interesting. Ben, did you change any replication-related GUCs that
could influence that? Say, wal_receiver_timeout, hot_standby_feedback
or max_standby_streaming_delay?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 04:28 Ben Chobot <bench@silentmedia.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 25+ messages in thread
From: Ben Chobot @ 2022-02-09 04:28 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Michael Paquier wrote on 2/8/22 5:35 PM:
> On Tue, Feb 08, 2022 at 01:23:34PM -0800, Peter Geoghegan wrote:
>> I find the idea that we'd fail to WAL-log information that is needed
>> during Hot Standby (to prevent this race condition) plausible.
>> Michael?
> Yeah, REINDEX relies on some existing index definition, so it feels
> like we are missing a piece related to invalid indexes in all that. A
> main difference is the lock level, as exclusive locks are getting
> logged so the standby can react and wait on that. The 30-minute mark
> is interesting. Ben, did you change any replication-related GUCs that
> could influence that? Say, wal_receiver_timeout, hot_standby_feedback
> or max_standby_streaming_delay?
Oh, to be clear, the 30 minute mark is more "the loop has always failed
this far into it" and sometimes that's 5 minutes and sometimes it's
more, but I've never seen it take more than somewhere in the 20s. I was
thinking it was just because of the race condition, but, to answer your
question, yes, we have tuned some replication parameters. Here are the
ones you asked about; did you want to see the value of any others?
=# show wal_receiver_timeout ;
wal_receiver_timeout
──────────────────────
1min
(1 row)
04:26:27 db: postgres@postgres, pid:29507
=# show hot_standby_feedback ;
hot_standby_feedback
──────────────────────
on
(1 row)
04:26:40 db: postgres@postgres, pid:29507
=# show max_standby_streaming_delay ;
max_standby_streaming_delay
─────────────────────────────
10min
(1 row)
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 05:25 Andrey Borodin <x4mmm@yandex-team.ru>
parent: PG Bug reporting form <noreply@postgresql.org>
1 sibling, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-02-09 05:25 UTC (permalink / raw)
To: bench@silentmedia.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 9 февр. 2022 г., в 01:41, PG Bug reporting form <noreply@postgresql.org> написал(а):
>
> PostgreSQL version: 12.9
I thought we fixed this in 12.9. But apparently not.
Gitlab was observing similar problem on 12.7 here [0]. And I was convinced that it's manifestation of relcache bug fixed in fdd965d07 [1]. I think we could adapt current amcheck tests for replication checks too. It cannot do heapallindexed on standby, but probably can catch relcache bugs.
Best regards, Andrey Borodin.
[0] https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6120#note_796846035
[1] https://github.com/postgres/postgres/commit/fdd965d074d46765c295223b119ca437dbcac973
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 05:30 Peter Geoghegan <pg@bowt.ie>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Peter Geoghegan @ 2022-02-09 05:30 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Feb 8, 2022 at 9:25 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> It cannot do heapallindexed on standby, but probably can catch relcache bugs.
Why not? In general heapallindexed verification works just fine on a standby.
You may be thinking of the extra verification options that are applied
to the index structure itself when the user opts to call
bt_index_parent_check() (rather than bt_index_check). But that's
unrelated.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 06:52 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-02-09 06:52 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> 9 февр. 2022 г., в 10:30, Peter Geoghegan <pg@bowt.ie> написал(а):
>
> On Tue, Feb 8, 2022 at 9:25 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>> It cannot do heapallindexed on standby, but probably can catch relcache bugs.
>
> Why not? In general heapallindexed verification works just fine on a standby.
Yes, but it acquires big ShareLock on relation. Probability of unusual interference with startup redo sequences becomes much smaller.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-09 07:13 Peter Geoghegan <pg@bowt.ie>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Peter Geoghegan @ 2022-02-09 07:13 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Feb 8, 2022 at 10:52 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> Yes, but it acquires big ShareLock on relation. Probability of unusual interference with startup redo sequences becomes much smaller.
Actually, calling bt_index_check() + heapallindexed acquires only an
AccessShareLock on both the table and the index.
Again, it sounds like you're talking about bt_index_parent_check()
(with or without heapallindexed). That does acquire a ShareLock, which
will just throw an error on a standby.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-10 16:52 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-02-10 16:52 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Ben Chobot <bench@silentmedia.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On 9 Feb 2022, at 12:13, Peter Geoghegan <pg@bowt.ie> wrote:
>
> On Tue, Feb 8, 2022 at 10:52 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>> Yes, but it acquires big ShareLock on relation. Probability of unusual interference with startup redo sequences becomes much smaller.
>
> Actually, calling bt_index_check() + heapallindexed acquires only an
> AccessShareLock on both the table and the index.
>
> Again, it sounds like you're talking about bt_index_parent_check()
> (with or without heapallindexed). That does acquire a ShareLock, which
> will just throw an error on a standby.
You are right.
So, I’ve composed dirty test by reverting 7f580aa5d [0] to resurrect pgbench_background().
Now I observe in standby log:
2022-02-10 21:46:38.909 +05 [84272] 004_cic_standby.pl LOG: statement: SELECT bt_index_check('idx',false);
2022-02-10 21:46:38.910 +05 [84272] 004_cic_standby.pl ERROR: cannot check index "idx_ccold"
2022-02-10 21:46:38.910 +05 [84272] 004_cic_standby.pl DETAIL: Index is not valid.
2022-02-10 21:46:38.910 +05 [84272] 004_cic_standby.pl STATEMENT: SELECT bt_index_check('idx',false);
I think it’s somewhat unexpected. But is it a real problem?
Thanks!
[0] https://github.com/postgres/postgres/commit/7f580aa5d88a9b03d66fcb9a1d7c4fcd69d9e126
Attachments:
0001-Try-to-check-indexes-on-standby.patch (application/octet-stream, 6.9K) download
From ef3d301697afabc3ad121dccbbcaba0149d1df25 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <x4mmm@flight.local>
Date: Thu, 10 Feb 2022 21:47:44 +0500
Subject: [PATCH] Try to check indexes on standby
---
contrib/amcheck/t/002_cic.pl | 48 +++++++++++++--------
contrib/amcheck/t/003_cic_2pc.pl | 53 +++++++++++++-----------
src/test/perl/PostgreSQL/Test/Cluster.pm | 48 +++++++++++++++++++++
3 files changed, 108 insertions(+), 41 deletions(-)
diff --git a/contrib/amcheck/t/002_cic.pl b/contrib/amcheck/t/002_cic.pl
index f6b91e89ad..58689423c9 100644
--- a/contrib/amcheck/t/002_cic.pl
+++ b/contrib/amcheck/t/002_cic.pl
@@ -9,7 +9,7 @@ use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 3;
+use Test::More tests => 4;
my ($node, $result);
@@ -25,18 +25,32 @@ $node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
$node->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
#
-# Stress CIC with pgbench.
-#
-# pgbench might try to launch more than one instance of the CIC
-# transaction concurrently. That would deadlock, so use an advisory
-# lock to ensure only one CIC runs at a time.
+# Stress CIC with pgbench
#
+
+# Run background pgbench with CIC. We cannot mix-in this script into single
+# pgbench: CIC will deadlock with itself occasionally.
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench(
+ '--no-vacuum --client=1 --transactions=200',
+ {
+ '002_pgbench_concurrent_cic' => q(
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
+# Run pgbench.
$node->pgbench(
- '--no-vacuum --client=5 --transactions=100',
+ '--no-vacuum --client=5 --transactions=200',
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs and CIC',
+ 'concurrent INSERTs',
{
'002_pgbench_concurrent_transaction' => q(
BEGIN;
@@ -48,17 +62,17 @@ $node->pgbench(
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
COMMIT;
- ),
- '002_pgbench_concurrent_cic' => q(
- SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
- \if :gotlock
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- SELECT bt_index_check('idx',true);
- SELECT pg_advisory_unlock(42);
- \endif
)
});
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
$node->stop;
done_testing();
diff --git a/contrib/amcheck/t/003_cic_2pc.pl b/contrib/amcheck/t/003_cic_2pc.pl
index 2f804efb84..e78de60e64 100644
--- a/contrib/amcheck/t/003_cic_2pc.pl
+++ b/contrib/amcheck/t/003_cic_2pc.pl
@@ -9,7 +9,7 @@ use Config;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 5;
+use Test::More tests => 6;
local $TODO = 'filesystem bug' if PostgreSQL::Test::Utils::has_wal_read_bug;
@@ -134,52 +134,57 @@ is($result, '0', 'bt_index_check after 2PC and restart');
#
# Stress CIC+2PC with pgbench
#
-# pgbench might try to launch more than one instance of the CIC
-# transaction concurrently. That would deadlock, so use an advisory
-# lock to ensure only one CIC runs at a time.
# Fix broken index first
$node->safe_psql('postgres', q(REINDEX TABLE tbl;));
+# Run background pgbench with CIC. We cannot mix-in this script into single
+# pgbench: CIC will deadlock with itself occasionally.
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node->background_pgbench(
+ '--no-vacuum --client=1 --transactions=100',
+ {
+ '002_pgbench_concurrent_cic' => q(
+ DROP INDEX CONCURRENTLY idx;
+ CREATE INDEX CONCURRENTLY idx ON tbl(i);
+ SELECT bt_index_check('idx',true);
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
# Run pgbench.
$node->pgbench(
'--no-vacuum --client=5 --transactions=100',
0,
[qr{actually processed}],
[qr{^$}],
- 'concurrent INSERTs w/ 2PC and CIC',
+ 'concurrent INSERTs w/ 2PC',
{
- '003_pgbench_concurrent_2pc' => q(
+ '002_pgbench_concurrent_2pc' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
),
- '003_pgbench_concurrent_2pc_savepoint' => q(
+ '002_pgbench_concurrent_2pc_savepoint' => q(
BEGIN;
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
- ),
- '003_pgbench_concurrent_cic' => q(
- SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
- \if :gotlock
- DROP INDEX CONCURRENTLY idx;
- CREATE INDEX CONCURRENTLY idx ON tbl(i);
- SELECT bt_index_check('idx',true);
- SELECT pg_advisory_unlock(42);
- \endif
- ),
- '004_pgbench_concurrent_ric' => q(
- SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
- \if :gotlock
- REINDEX INDEX CONCURRENTLY idx;
- SELECT bt_index_check('idx',true);
- SELECT pg_advisory_unlock(42);
- \endif
)
});
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+$result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with CIC works");
+
+# done
$node->stop;
done_testing();
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 41beadff9d..8041092edb 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2142,6 +2142,54 @@ sub pgbench
=pod
+=item $node->background_pgbench($opts, $files, \$stdout, $timer) => harness
+
+Invoke B<pgbench> and return an IPC::Run harness object. The process's stdin
+is empty, and its stdout and stderr go to the $stdout scalar reference. This
+allows the caller to act on other parts of the system while B<pgbench> is
+running. Errors from B<pgbench> are the caller's problem.
+
+The specified timer object is attached to the harness, as well. It's caller's
+responsibility to select the timeout length, and to restart the timer after
+each command if the timeout is per-command.
+
+Be sure to "finish" the harness when done with it.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=back
+
+=cut
+
+sub background_pgbench
+{
+ my ($self, $opts, $files, $stdout, $timer) = @_;
+
+ my @cmd =
+ ('pgbench', split(/\s+/, $opts), $self->_pgbench_make_files($files));
+
+ local %ENV = $self->_get_env();
+
+ my $stdin = "";
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
+ $timer;
+
+ return $harness;
+}
+
+=pod
+
=item $node->connect_ok($connstr, $test_name, %params)
Attempt a connection with a custom connection string. This is expected
--
2.32.0 (Apple Git-132)
=
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-10 17:18 Ben Chobot <bench@silentmedia.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Ben Chobot @ 2022-02-10 17:18 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Andrey Borodin wrote on 2/10/22 8:52 AM:
> I think it’s somewhat unexpected. But is it a real problem?
If you're asking me, then I can't speak to your standby log, but I'd say
the bug I raised is a problem, yes. :)
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-10 17:41 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Ben Chobot <bench@silentmedia.com>
0 siblings, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-02-10 17:41 UTC (permalink / raw)
To: Ben Chobot <bench@silentmedia.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On 10 Feb 2022, at 22:18, Ben Chobot <bench@silentmedia.com> wrote:
>
> Andrey Borodin wrote on 2/10/22 8:52 AM:
>> I think it’s somewhat unexpected. But is it a real problem?
>
> If you're asking me, then I can't speak to your standby log, but I'd say the bug I raised is a problem, yes. :)
Well, not exactly. I had attached the patch with the test that reproduces some problem with dropped index on standby.
In this test we have a standby and check query on it at some point finds out that it’s working with invalid index. I’m not sure that this is reproduction of your exact problem.
I see that we are swapping indexes on primary in a following way:
index_concurrently_swap(newidx->indexId, oldidx->indexId, oldName); // WAL-logged
CacheInvalidateRelcacheByRelid(oldidx->tableId); // Not WAL-logged
CommitTransactionCommand(); // WAL-logged
But I do not know is it expected to work on standby, since relcache invalidation is not logged. Maybe Peter or someone more experienced than me can explain how this is expected to work.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-11 00:12 Andres Freund <andres@anarazel.de>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 25+ messages in thread
From: Andres Freund @ 2022-02-11 00:12 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-02-10 22:41:00 +0500, Andrey Borodin wrote:
> index_concurrently_swap(newidx->indexId, oldidx->indexId, oldName); // WAL-logged
> CacheInvalidateRelcacheByRelid(oldidx->tableId); // Not WAL-logged
> CommitTransactionCommand(); // WAL-logged
>
> But I do not know is it expected to work on standby, since relcache
> invalidation is not logged. Maybe Peter or someone more experienced than me
> can explain how this is expected to work.
What makes you say that CacheInvalidateRelcacheByRelid isn't WAL logged? It
should be emitted at the commit, like other invalidation messages?
Here's the WAL for a REINDEX CONCURRENTLY, leaving out the actual catalog
changes (i.e. heap/heap2/btree rmgr):
rmgr: Storage len (rec/tot): 42/ 42, tx: 0, lsn: 0/0155F3F8, prev 0/0155F3B8, desc: CREATE base/5/16399
rmgr: Standby len (rec/tot): 42/ 42, tx: 731, lsn: 0/0155F428, prev 0/0155F3F8, desc: LOCK xid 731 db 5 rel 16399
rmgr: Transaction len (rec/tot): 194/ 194, tx: 731, lsn: 0/0155F960, prev 0/0155F918, desc: COMMIT 2022-02-10 15:53:56.173778 PST; inval msgs: catcache 53 catcache 52 catcache 7 catcache 6 catcache 32 relcache 16399 relcache 16392 snapshot 2608 relcache 16392
rmgr: Transaction len (rec/tot): 146/ 146, tx: 732, lsn: 0/01561BB0, prev 0/01561B70, desc: COMMIT 2022-02-10 15:53:56.192747 PST; inval msgs: catcache 53 catcache 52 catcache 32 relcache 16392 relcache 16399 relcache 16399
It might be worth writing a test that replays WAL records one-by-one, and that
checks at which record things start to break. If the problem is not
reproducible that way, we have a problem "within" a single WAL record
replay. If it is reproducible, it's likely a problem in the way the primary
generates WAL.
That would be generally very useful tooling for detailed testing of sequences
of WAL records.
I'm pretty sure the problem is on the primary. Looking through
ReindexRelationConcurrently() I think I found at least two problems:
1) We don't WAL log snapshot conflicts, afaict (i.e. the
WaitForOlderSnapshots() in phase 3). Which I think means that the new index
can end up being used "too soon" in pre-existing snapshots on the standby.
I don't think this is the problem this thread is about, but it's definitely a
problem.
2) WaitForLockersMultiple() in phase 5 / 6 isn't WAL logged. Without waiting
for sessions to see the results of Phase 4, 5, we can mark the index as dead
(phase 5) and drop it (phase 6), while there are ongoing accesses.
I think this is the likely cause of the reported bug.
Both seems like a significant issue for several of the CONCURRENTLY commands,
not just REINDEX.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-11 01:38 Michael Paquier <michael@paquier.xyz>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 25+ messages in thread
From: Michael Paquier @ 2022-02-11 01:38 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Feb 10, 2022 at 04:12:40PM -0800, Andres Freund wrote:
> I'm pretty sure the problem is on the primary. Looking through
> ReindexRelationConcurrently() I think I found at least two problems:
>
> 1) We don't WAL log snapshot conflicts, afaict (i.e. the
> WaitForOlderSnapshots() in phase 3). Which I think means that the new index
> can end up being used "too soon" in pre-existing snapshots on the standby.
>
> I don't think this is the problem this thread is about, but it's definitely a
> problem.
>
> 2) WaitForLockersMultiple() in phase 5 / 6 isn't WAL logged. Without waiting
> for sessions to see the results of Phase 4, 5, we can mark the index as dead
> (phase 5) and drop it (phase 6), while there are ongoing accesses.
>
> I think this is the likely cause of the reported bug.
Yep, I was planning to play with this problem from next week, FWIW,
just lacked time/energy to do so. And the release was shipping
anyway, so there is plenty of time.
My impression is that we don't really need to change the WAL format
based on the existing APIs we already have, or that in the worst case
it would be possible to make things backward-compatible enough that it
would not be a worry as long as the standbys are updated before the
primaries.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-11 02:18 Andres Freund <andres@anarazel.de>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 25+ messages in thread
From: Andres Freund @ 2022-02-11 02:18 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-02-11 10:38:34 +0900, Michael Paquier wrote:
> My impression is that we don't really need to change the WAL format
> based on the existing APIs we already have, or that in the worst case
> it would be possible to make things backward-compatible enough that it
> would not be a worry as long as the standbys are updated before the
> primaries.
I think it's a question of how "concurrently" we want concurrently to be on
hot standby nodes.
If we are OK with "not at all", we can just lock an AEL at the end of
WaitForLockersMultiple(). It's a bit harder to cause the necessary snapshot
conflict, but we could e.g. extend xl_running_xacts and use the record length
to keep both older primary -> newer replica, and the reverse working.
If we want to actually make concurrently work concurrently on the replica,
we'd have to work harder. That doesn't strike me as feasible for backpatching.
The most important bit bit would be to make WaitForOlderSnapshots() wait for
the xmin of hot_standby_feedback walsenders and/or physical slot xmin
horizons. For phase 5 & 6 we would probably have to do something like waiting
for snapshot conflicts vs walsenders/slots in addition to the
WaitForLockersMultiple() for conflicts on the primary.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-02-12 17:03 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-02-12 17:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On 11 Feb 2022, at 05:12, Andres Freund <andres@anarazel.de> wrote:
>
> On 2022-02-10 22:41:00 +0500, Andrey Borodin wrote:
>> index_concurrently_swap(newidx->indexId, oldidx->indexId, oldName); // WAL-logged
>> CacheInvalidateRelcacheByRelid(oldidx->tableId); // Not WAL-logged
>> CommitTransactionCommand(); // WAL-logged
>>
>> But I do not know is it expected to work on standby, since relcache
>> invalidation is not logged. Maybe Peter or someone more experienced than me
>> can explain how this is expected to work.
>
> What makes you say that CacheInvalidateRelcacheByRelid isn't WAL logged? It
> should be emitted at the commit, like other invalidation messages?
>
Thank you for the explanation!
FWIW here’s more clean TAP-test reproduction. Sometimes I see
2022-02-12 21:59:02.786 +05 [19629] 004_cic_standby.pl ERROR: could not open relation with OID 16401
2022-02-12 21:59:02.786 +05 [19629] 004_cic_standby.pl STATEMENT: SELECT bt_index_check('idx',true);
In tmp_check/log/004_cic_standby_standby_1.log after running amcheck tests. Exactly as per initial report.
Best regards, Andrey Borodin.
Attachments:
0001-Test-B-tree-index-on-standby-during-REINDEX-CONCURRE.patch (application/octet-stream, 5.4K) download
From 3c3f0dedc72834c5ea52f9e490eff58ea602680f Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <x4mmm@flight.local>
Date: Sat, 12 Feb 2022 13:51:06 +0500
Subject: [PATCH] Test B-tree index on standby during REINDEX CONCURRENTLY
Currently we do not transfer WaitForLockersMultiple() information
to standby. This may lead to improper index use.
---
contrib/amcheck/t/004_cic_standby.pl | 102 +++++++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 48 +++++++++++
2 files changed, 150 insertions(+)
create mode 100644 contrib/amcheck/t/004_cic_standby.pl
diff --git a/contrib/amcheck/t/004_cic_standby.pl b/contrib/amcheck/t/004_cic_standby.pl
new file mode 100644
index 0000000000..0c644f0fb8
--- /dev/null
+++ b/contrib/amcheck/t/004_cic_standby.pl
@@ -0,0 +1,102 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Minimal test testing REINDEX CONCURRENTLY with streaming replication
+use strict;
+use warnings;
+use Config;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+# A specific role is created for replication purposes
+$node_primary->init(
+ allows_streaming => 1,
+ auth_extra => [ '--create-role', 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', 'lock_timeout = 180000');
+$node_primary->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->start;
+
+# Create some content on primary and check its presence in standby nodes
+$node_primary->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node_primary->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node_primary->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
+
+# Wait for standbys to catch up
+my $primary_lsn = $node_primary->lsn('write');
+$node_primary->wait_for_catchup($node_standby_1, 'replay', $primary_lsn);
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with bt_index_check on standby
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node_standby_1->background_pgbench(
+ '--no-vacuum --client=1 --transactions=2000',
+ {
+ '004_pgbench_standby_check_1' => q(
+ SELECT bt_index_check('idx',true);
+ ),
+ '004_pgbench_standby_check_2' => q(
+ SELECT bt_index_check('idx',false);
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
+# Run pgbench with data data manipulations and REINDEX on primary.
+# pgbench might try to launch more than one instance of the RIC
+# transaction concurrently. That would deadlock, so use an advisory
+# lock to ensure only one CIC runs at a time.
+$node_primary->pgbench(
+ '--no-vacuum --client=5 --transactions=2000',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent INSERTs and CIC',
+ {
+ '004_pgbench_concurrent_transaction' => q(
+ BEGIN;
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ ),
+ '004_pgbench_concurrent_transaction_savepoints' => q(
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO tbl VALUES(0);
+ COMMIT;
+ ),
+ '004_pgbench_concurrent_ric' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT pg_advisory_unlock(42);
+ \endif
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+my $result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with bt_index_check() on standby works");
+
+# done
+$node_primary->stop;
+$node_standby_1->stop;
+done_testing();
\ No newline at end of file
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 41beadff9d..8041092edb 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2142,6 +2142,54 @@ sub pgbench
=pod
+=item $node->background_pgbench($opts, $files, \$stdout, $timer) => harness
+
+Invoke B<pgbench> and return an IPC::Run harness object. The process's stdin
+is empty, and its stdout and stderr go to the $stdout scalar reference. This
+allows the caller to act on other parts of the system while B<pgbench> is
+running. Errors from B<pgbench> are the caller's problem.
+
+The specified timer object is attached to the harness, as well. It's caller's
+responsibility to select the timeout length, and to restart the timer after
+each command if the timeout is per-command.
+
+Be sure to "finish" the harness when done with it.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=back
+
+=cut
+
+sub background_pgbench
+{
+ my ($self, $opts, $files, $stdout, $timer) = @_;
+
+ my @cmd =
+ ('pgbench', split(/\s+/, $opts), $self->_pgbench_make_files($files));
+
+ local %ENV = $self->_get_env();
+
+ my $stdin = "";
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
+ $timer;
+
+ return $harness;
+}
+
+=pod
+
=item $node->connect_ok($connstr, $test_name, %params)
Attempt a connection with a custom connection string. This is expected
--
2.32.0 (Apple Git-132)
=
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-03-08 15:44 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-03-08 15:44 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On 12 Feb 2022, at 20:03, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> Thank you for the explanation!
> FWIW here’s more clean TAP-test reproduction. Sometimes I see
> 2022-02-12 21:59:02.786 +05 [19629] 004_cic_standby.pl ERROR: could not open relation with OID 16401
> 2022-02-12 21:59:02.786 +05 [19629] 004_cic_standby.pl STATEMENT: SELECT bt_index_check('idx',true);
>
> In tmp_check/log/004_cic_standby_standby_1.log after running amcheck tests. Exactly as per initial report.
I tried to dig into this again. And once again I simply cannot understand how it is supposed to work... I'd appreciate if someone explained it to me.
When I do SELECT bt_index_check('idx',true) in fact I'm doing following:
1. regclassin() lookups 'idx', converts it into oid without taking any lock
2. bt_index_check() gets this oid and lookups index again.
Do we rely on catalog snapshot here? Or how do we know that this oid is still of the index named 'idx' on standby? When backend will understand that this oid is no more of the interest and get an error somehow? I think it must happen during index_open().
BTW is anyone working on this bug?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-20 07:14 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Michael Paquier @ 2022-04-20 07:14 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Mar 08, 2022 at 06:44:31PM +0300, Andrey Borodin wrote:
> I tried to dig into this again. And once again I simply cannot
> understand how it is supposed to work... I'd appreciate if someone
> explained it to me.
> When I do SELECT bt_index_check('idx',true) in fact I'm doing following:
> 1. regclassin() lookups 'idx', converts it into oid without taking any lock
> 2. bt_index_check() gets this oid and lookups index again.
> Do we rely on catalog snapshot here? Or how do we know that this oid
> is still of the index named 'idx' on standby? When backend will
> understand that this oid is no more of the interest and get an error
> somehow? I think it must happen during index_open().
Yeah, I can see that, and I think that it is a second issue, rather
independent of the cache lookup errors that we are seeing on a
standby. As far as I have tested, it is possible to see this error as
well with a sequence of CREATE/DROP INDEX CONCURRENTLY run repeatedly
on a primary with the standby doing a scan of pg_index, like that for
example in a tight loop:
SELECT bt_index_check(indexrelid) FROM pg_index WHERE indisvalid AND
indexrelid > 16000;
It is a matter of seconds to see bt_index_check() complain that one of
the so-said index is not valid, but the scan of pg_index was told to
not select such an index, so we are missing something with the
relation cache.
> BTW is anyone working on this bug?
It took me some time to come back to this thread, but I do now that we
are done with the commit fest business.
Speaking of the cache errors when working directly on the relations
rebuilt, I got a patch that seems to work properly, with a mix of
standby snapshots and AELs logged. I have been running a set of
REINDEX INDEX/TABLE CONCURRENTLY on the primary through more than 100k
relfilenodes without the standby complaining, and I should be able to
get this one fixed before the next minor release.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-20 18:23 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-04-20 18:23 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On 20 Apr 2022, at 12:14, Michael Paquier <michael@paquier.xyz> wrote:
>
> I got a patch that seems to work properly,
Cool! Can you please share it?
> with a mix of
> standby snapshots and AELs logged.
I did not succeed with my tests on this way. But probably I was doing something wrong. Or, perhaps, these tests were encountering second problem.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-21 03:54 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 25+ messages in thread
From: Michael Paquier @ 2022-04-21 03:54 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Apr 20, 2022 at 11:23:43PM +0500, Andrey Borodin wrote:
> Cool! Can you please share it?
Sure. I am finishing for this part with the attached, that would be
backpatchable. Based on the analysis of upthread, I have stuck with
logging standby snapshots once we are done in WaitForOlderSnapshots()
so as we make sure that a standby does not attempt to look at the data
of any running transactions it should wait for and the log of AELs at
the end of WaitForLockersMultiple() to avoid the access of the lockers
too early. I have to admit that a direct use of
LogAccessExclusiveLock() feels a bit funky but we cannot use
LockAcquire() AFAIK, and that should be fine with all the lock tags at
hand.
I have stressed quite a lot a primary/standby setup with flavors of
the tests from Ben upthread, and I am not seeing the cache problems
anymore on the standby, with concurrent CREATE/DROP INDEX or REINDEX
sequences.
> I did not succeed with my tests on this way. But probably I was
> doing something wrong. Or, perhaps, these tests were encountering
> second problem.
There is a second thing going on with amcheck here. REINDEX
CONCURRENTLY in isolation is working correctly when you run
bt_index_check() in parallel of it (aka attach gdb to the backend
process running REINDEX CONCURRENTLY and run bt_index_check()
repeatedly in parallel at the various steps with a \watch or a new
connection each time), but depending on your transaction order you may
finish with an invalid index given as input of
bt_index_check_internal(), roughly:
- REINDEX CONCURRENTLY begins, goes up to the transaction doing the
swap where the old index is marked as indisvalid.
- Execute bt_index_check(), that refers to the old index OID, still
valid at the moment where amcheck begins checking this index, before
the Relation of the parent table is opened.
- The transaction doing the swap commits, making visible that
indisvalid is false for the old index.
- bt_index_check() continues, opens the parent Relation, and complains
about an invalid index.
amcheck could be made more robust here, by calling
RelationGetIndexList() after opening the Relation of the parent to
check if it is still listed in the returned list as it would look at
the relcache and discard any invalid indexes on the way. Returning an
error for an invalid index seems less helpful to me here than doing
nothing, particularly if you begin doing a full check of the indexes
based on the contents of pg_index.
--
Michael
Attachments:
reindex-standby-fix.patch (text/x-diff, 1.7K) download
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index cd30f15eba..704d8f3744 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -475,6 +475,16 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
if (progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
}
+
+ /*
+ * Take a snapshot of running transactions and write this to WAL. With
+ * this information, a standby is able to know that it should not access
+ * too soon any information we expect to wait for with a concurrent build.
+ *
+ * Skip the log of this information if disabled.
+ */
+ if (XLogStandbyInfoActive())
+ LogStandbySnapshot();
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 1543da6162..81f0e7c88c 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -974,6 +974,25 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_multi_param(3, index, values);
}
+ /*
+ * Now that the wait is done, log a set of access exclusive locks on
+ * the relation locktags we waited for. This level of lock is higher
+ * than what a standby should need to be able to look at the catalog
+ * information of the objects involved. This ensures to block any
+ * attempt to the objects that it will until the transaction running
+ * this routine is committed at recovery.
+ */
+ foreach(lc, locktags)
+ {
+ LOCKTAG *locktag = lfirst(lc);
+
+ if (locktag->locktag_type != LOCKTAG_RELATION)
+ continue;
+
+ LogAccessExclusiveLock(locktag->locktag_field1, /* dbId */
+ locktag->locktag_field2); /* relId */
+ }
+
list_free_deep(holders);
}
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-21 16:17 Andres Freund <andres@anarazel.de>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 25+ messages in thread
From: Andres Freund @ 2022-04-21 16:17 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-04-21 12:54:00 +0900, Michael Paquier wrote:
> Based on the analysis of upthread, I have stuck with logging standby
> snapshots once we are done in WaitForOlderSnapshots() so as we make sure
> that a standby does not attempt to look at the data of any running
> transactions it should wait for and the log of AELs at the end of
> WaitForLockersMultiple() to avoid the access of the lockers too early.
Why is this necessary? The comments in WaitForOlderSnapshots() don't really
explain it. This is pretty darn expensive.
> amcheck could be made more robust here, by calling
> RelationGetIndexList() after opening the Relation of the parent to
> check if it is still listed in the returned list as it would look at
> the relcache and discard any invalid indexes on the way.
That seems like a weird approach. Why can't the check just be done on the
relcache entry of the index itself? If that doesn't work, something is still
broken in cache invalidation.
> diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
> index cd30f15eba..704d8f3744 100644
> --- a/src/backend/commands/indexcmds.c
> +++ b/src/backend/commands/indexcmds.c
> @@ -475,6 +475,16 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
> if (progress)
> pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, i + 1);
> }
> +
> + /*
> + * Take a snapshot of running transactions and write this to WAL. With
> + * this information, a standby is able to know that it should not access
> + * too soon any information we expect to wait for with a concurrent build.
> + *
> + * Skip the log of this information if disabled.
> + */
> + if (XLogStandbyInfoActive())
> + LogStandbySnapshot();
> }
"should not access too soon" - no idea what that means / guarantees.
The "Skip the log" bit is redundant with the code.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-23 06:02 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 25+ messages in thread
From: Andrey Borodin @ 2022-04-23 06:02 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Here are few more observations from me.
> 21 апр. 2022 г., в 08:54, Michael Paquier <michael@paquier.xyz> написал(а):
>
> There is a second thing going on with amcheck here.
I've removed amcheck from the test and replaced it with a simple IndexOnlyScan. I'm not proposing to commit this test anyway, so I left it in amcheck.
The test reliably fails in 30s or earlier.
2022-04-23 10:51:06.872 +05 [36192] 004_cic_standby.pl LOG: statement: select i from tbl where i = 236116;
2022-04-23 10:51:06.890 +05 [36192] 004_cic_standby.pl ERROR: could not open relation with OID 16671
2022-04-23 10:51:06.890 +05 [36192] 004_cic_standby.pl STATEMENT: select i from tbl where i = 236116;
Adding AEL's seem to solve the problem. If I run test further, approximately after 40s it will fail with:
2022-04-23 10:47:47.669 +05 [34225] 004_cic_standby.pl LOG: statement: select i from tbl where i = 310032;
2022-04-23 10:47:47.670 +05 [34225] 004_cic_standby.pl FATAL: terminating connection due to conflict with recovery
2022-04-23 10:47:47.670 +05 [34225] 004_cic_standby.pl DETAIL: User was holding a relation lock for too long.
Is it kind of deadlock?
I did not observe any changes in the behaviour of the test adding LogStandbySnapshot(). Can we trigger its necessity somehow? Probably, by solving "conflict with recovery" issue and running test for a longer time?
Thanks!
Best regards, Andrey Borodin.
Attachments:
v2-0001-Test-B-tree-index-on-standby-during-REINDEX-CONCU.patch (application/octet-stream, 5.3K) download
From 058794eda6e5156d4b6f2080f76515060f04d541 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <x4mmm@flight.local>
Date: Sat, 12 Feb 2022 13:51:06 +0500
Subject: [PATCH v2] Test B-tree index on standby during REINDEX CONCURRENTLY
Currently we do not transfer WaitForLockersMultiple() information
to standby. This may lead to improper index use.
---
contrib/amcheck/t/004_cic_standby.pl | 96 ++++++++++++++++++++++++
src/test/perl/PostgreSQL/Test/Cluster.pm | 48 ++++++++++++
2 files changed, 144 insertions(+)
create mode 100644 contrib/amcheck/t/004_cic_standby.pl
diff --git a/contrib/amcheck/t/004_cic_standby.pl b/contrib/amcheck/t/004_cic_standby.pl
new file mode 100644
index 0000000000..16b361cc64
--- /dev/null
+++ b/contrib/amcheck/t/004_cic_standby.pl
@@ -0,0 +1,96 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Minimal test testing REINDEX CONCURRENTLY with streaming replication
+use strict;
+use warnings;
+use Config;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+
+# Initialize primary node
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+# A specific role is created for replication purposes
+$node_primary->init(
+ allows_streaming => 1,
+ auth_extra => [ '--create-role', 'repl_role' ]);
+$node_primary->append_conf('postgresql.conf', 'lock_timeout = 180000');
+$node_primary->start;
+my $backup_name = 'my_backup';
+
+# Take backup
+$node_primary->backup($backup_name);
+
+# Create streaming standby linking to primary
+my $node_standby_1 = PostgreSQL::Test::Cluster->new('standby_1');
+$node_standby_1->init_from_backup($node_primary, $backup_name,
+ has_streaming => 1);
+$node_standby_1->start;
+
+# Create some content on primary and check its presence in standby nodes
+$node_primary->safe_psql('postgres', q(CREATE EXTENSION amcheck));
+$node_primary->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
+$node_primary->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
+
+# Wait for standbys to catch up
+my $primary_lsn = $node_primary->lsn('write');
+$node_primary->wait_for_catchup($node_standby_1, 'replay', $primary_lsn);
+
+#
+# Stress CIC with pgbench
+#
+
+# Run background pgbench with bt_index_check on standby
+my $pgbench_out = '';
+my $pgbench_timer = IPC::Run::timeout(180);
+my $pgbench_h = $node_standby_1->background_pgbench(
+ '--no-vacuum --client=1 --time=30',
+ {
+ '004_pgbench_standby_check_1' => q(
+ set enable_bitmapscan to off;
+ set enable_seqscan to off;
+ \set id random(1, 1000000)
+ select i from tbl where i = :id;
+ )
+ },
+ \$pgbench_out,
+ $pgbench_timer);
+
+# Run pgbench with data data manipulations and REINDEX on primary.
+# pgbench might try to launch more than one instance of the RIC
+# transaction concurrently. That would deadlock, so use an advisory
+# lock to ensure only one CIC runs at a time.
+$node_primary->pgbench(
+ '--no-vacuum --client=5 --time=30',
+ 0,
+ [qr{actually processed}],
+ [qr{^$}],
+ 'concurrent INSERTs and CIC',
+ {
+ '004_pgbench_concurrent_transaction' => q(
+ BEGIN;
+ INSERT INTO tbl VALUES((random()*1000000)::int);
+ COMMIT;
+ ),
+ '004_pgbench_concurrent_ric' => q(
+ SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
+ \if :gotlock
+ REINDEX INDEX CONCURRENTLY idx;
+ SELECT pg_advisory_unlock(42);
+ \endif
+ )
+ });
+
+$pgbench_h->pump_nb;
+$pgbench_h->finish();
+my $result =
+ ($Config{osname} eq "MSWin32")
+ ? ($pgbench_h->full_results)[0]
+ : $pgbench_h->result(0);
+is($result, 0, "pgbench with bt_index_check() on standby works");
+
+# done
+$node_primary->stop;
+$node_standby_1->stop;
+done_testing();
\ No newline at end of file
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 9a2ada0a10..c88a140334 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2099,6 +2099,54 @@ sub pgbench
=pod
+=item $node->background_pgbench($opts, $files, \$stdout, $timer) => harness
+
+Invoke B<pgbench> and return an IPC::Run harness object. The process's stdin
+is empty, and its stdout and stderr go to the $stdout scalar reference. This
+allows the caller to act on other parts of the system while B<pgbench> is
+running. Errors from B<pgbench> are the caller's problem.
+
+The specified timer object is attached to the harness, as well. It's caller's
+responsibility to select the timeout length, and to restart the timer after
+each command if the timeout is per-command.
+
+Be sure to "finish" the harness when done with it.
+
+=over
+
+=item $opts
+
+Options as a string to be split on spaces.
+
+=item $files
+
+Reference to filename/contents dictionary.
+
+=back
+
+=cut
+
+sub background_pgbench
+{
+ my ($self, $opts, $files, $stdout, $timer) = @_;
+
+ my @cmd =
+ ('pgbench', split(/\s+/, $opts), $self->_pgbench_make_files($files));
+
+ local %ENV = $self->_get_env();
+
+ my $stdin = "";
+ # IPC::Run would otherwise append to existing contents:
+ $$stdout = "" if ref($stdout);
+
+ my $harness = IPC::Run::start \@cmd, '<', \$stdin, '>', $stdout, '2>&1',
+ $timer;
+
+ return $harness;
+}
+
+=pod
+
=item $node->connect_ok($connstr, $test_name, %params)
Attempt a connection with a custom connection string. This is expected
--
2.24.3 (Apple Git-128)
=
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-24 03:03 Noah Misch <noah@leadboat.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 25+ messages in thread
From: Noah Misch @ 2022-04-24 03:03 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Drive-by comment:
On Sat, Apr 23, 2022 at 11:02:04AM +0500, Andrey Borodin wrote:
> I've removed amcheck from the test and replaced it with a simple IndexOnlyScan. I'm not proposing to commit this test anyway, so I left it in amcheck.
> The test reliably fails in 30s or earlier.
>
> 2022-04-23 10:51:06.872 +05 [36192] 004_cic_standby.pl LOG: statement: select i from tbl where i = 236116;
> 2022-04-23 10:51:06.890 +05 [36192] 004_cic_standby.pl ERROR: could not open relation with OID 16671
> 2022-04-23 10:51:06.890 +05 [36192] 004_cic_standby.pl STATEMENT: select i from tbl where i = 236116;
>
> Adding AEL's seem to solve the problem.
If this thread proceeds with the current LogAccessExclusiveLock() design, I
think it must include a way to disable the new behavior. That requirement
would apply to any design that delays standby WAL apply in a way that
hot_standby_feedback can't mitigate. We have the vacuum_truncate reloption
because AEL is disruptive to standby freshness. Adding a new kind of
unavoidable AEL would be similarly-painful.
It would nicer to fix this such that, with hot_standby_feedback, the fix
delays CIC on the primary instead of delaying apply on the standby. If that's
too hard, AEL plus option-to-disable sounds fine.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-25 04:47 Michael Paquier <michael@paquier.xyz>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 25+ messages in thread
From: Michael Paquier @ 2022-04-25 04:47 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Apr 21, 2022 at 09:17:10AM -0700, Andres Freund wrote:
> On 2022-04-21 12:54:00 +0900, Michael Paquier wrote:
>> Based on the analysis of upthread, I have stuck with logging standby
>> snapshots once we are done in WaitForOlderSnapshots() so as we make sure
>> that a standby does not attempt to look at the data of any running
>> transactions it should wait for and the log of AELs at the end of
>> WaitForLockersMultiple() to avoid the access of the lockers too early.
>
> Why is this necessary? The comments in WaitForOlderSnapshots() don't really
> explain it. This is pretty darn expensive.
I think that my brain here thought about the standby attempting to
access tuples that could have been deleted before the reference
snapshot was taken for the index validation in phase 3, and that
logging the current snapshot would help in detecting conflicts for
that. But that won't help much unless the conflicts themselves are
logged in some way. Is that right? It looks like this comes down to
log more information when we wait for some of the snapshots in
VirtualXactLock(). That would be new.
>> amcheck could be made more robust here, by calling
>> RelationGetIndexList() after opening the Relation of the parent to
>> check if it is still listed in the returned list as it would look at
>> the relcache and discard any invalid indexes on the way.
>
> That seems like a weird approach. Why can't the check just be done on the
> relcache entry of the index itself? If that doesn't work, something is still
> broken in cache invalidation.
As far as I looked at (aka checking step by step the concurrent
REINDEX and DROP INDEX flows), the cache looks fine with the
invalidations when the index's indisvalid is switched on/off (I am
taking into account the end of index_concurrently_build() that should
not need one).
FWIW, I would be tempted to do something like the attached for
amcheck, where we skip invalid indexes. Thoughts welcome.
--
Michael
Attachments:
amcheck-indisvalid.patch (text/x-diff, 1.1K) download
diff --git a/contrib/amcheck/verify_nbtree.c b/contrib/amcheck/verify_nbtree.c
index 70278c4f93..17e131bbe7 100644
--- a/contrib/amcheck/verify_nbtree.c
+++ b/contrib/amcheck/verify_nbtree.c
@@ -294,6 +294,14 @@ bt_index_check_internal(Oid indrelid, bool parentcheck, bool heapallindexed,
errmsg("could not open parent table of index \"%s\"",
RelationGetRelationName(indrel))));
+ /*
+ * An invalid index could be found here while a concurrent DROP INDEX
+ * or REINDEX is running in parallel. There is nothing interesting to
+ * do, so just leave.
+ */
+ if (!indrel->rd_index->indisvalid)
+ return;
+
/* Relation suitable for checking as B-Tree? */
btree_index_checkable(indrel);
@@ -361,13 +369,6 @@ btree_index_checkable(Relation rel)
errmsg("cannot access temporary tables of other sessions"),
errdetail("Index \"%s\" is associated with temporary relation.",
RelationGetRelationName(rel))));
-
- if (!rel->rd_index->indisvalid)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("cannot check index \"%s\"",
- RelationGetRelationName(rel)),
- errdetail("Index is not valid.")));
}
/*
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-25 05:11 Michael Paquier <michael@paquier.xyz>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 25+ messages in thread
From: Michael Paquier @ 2022-04-25 05:11 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Apr 23, 2022 at 08:03:47PM -0700, Noah Misch wrote:
> If this thread proceeds with the current LogAccessExclusiveLock() design, I
> think it must include a way to disable the new behavior. That requirement
> would apply to any design that delays standby WAL apply in a way that
> hot_standby_feedback can't mitigate. We have the vacuum_truncate reloption
> because AEL is disruptive to standby freshness. Adding a new kind of
> unavoidable AEL would be similarly-painful.
I am not sure that we should add more reloptions that bypass AEL
requirements FWIW, because the standby would still be opened to cache
errors if you don't log something, would it not?. I agree that
vacuum_truncate is handy, but there was a lot of drawback with it as
well when it was introduced as there were arguments that we'd better
remove the use of the AEL used for the end-of-vacuum truncation.
> It would nicer to fix this such that, with hot_standby_feedback, the fix
> delays CIC on the primary instead of delaying apply on the standby. If that's
> too hard, AEL plus option-to-disable sounds fine.
Perhaps. Still, that would be mostly what WaitForOlderSnapshots()
already does, except that we'd roughly just use the limitXmin from the
snapshot of the transaction currently running?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica
@ 2022-04-25 05:22 Noah Misch <noah@leadboat.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 25+ messages in thread
From: Noah Misch @ 2022-04-25 05:22 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Andres Freund <andres@anarazel.de>; Ben Chobot <bench@silentmedia.com>; Peter Geoghegan <pg@bowt.ie>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Apr 25, 2022 at 02:11:26PM +0900, Michael Paquier wrote:
> On Sat, Apr 23, 2022 at 08:03:47PM -0700, Noah Misch wrote:
> > If this thread proceeds with the current LogAccessExclusiveLock() design, I
> > think it must include a way to disable the new behavior. That requirement
> > would apply to any design that delays standby WAL apply in a way that
> > hot_standby_feedback can't mitigate. We have the vacuum_truncate reloption
> > because AEL is disruptive to standby freshness. Adding a new kind of
> > unavoidable AEL would be similarly-painful.
>
> I am not sure that we should add more reloptions that bypass AEL
> requirements FWIW, because the standby would still be opened to cache
> errors if you don't log something, would it not?
Correct. Even so, a mandatory AEL when replaying CIC would be worse.
> > It would nicer to fix this such that, with hot_standby_feedback, the fix
> > delays CIC on the primary instead of delaying apply on the standby. If that's
> > too hard, AEL plus option-to-disable sounds fine.
>
> Perhaps. Still, that would be mostly what WaitForOlderSnapshots()
> already does, except that we'd roughly just use the limitXmin from the
> snapshot of the transaction currently running?
I don't know.
^ permalink raw reply [nested|flat] 25+ messages in thread
end of thread, other threads:[~2022-04-25 05:22 UTC | newest]
Thread overview: 25+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-08 20:41 BUG #17401: REINDEX TABLE CONCURRENTLY creates a race condition on a streaming replica PG Bug reporting form <noreply@postgresql.org>
2022-02-08 21:23 ` Peter Geoghegan <pg@bowt.ie>
2022-02-09 01:35 ` Michael Paquier <michael@paquier.xyz>
2022-02-09 04:28 ` Ben Chobot <bench@silentmedia.com>
2022-02-09 05:25 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-02-09 05:30 ` Peter Geoghegan <pg@bowt.ie>
2022-02-09 06:52 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-02-09 07:13 ` Peter Geoghegan <pg@bowt.ie>
2022-02-10 16:52 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-02-10 17:18 ` Ben Chobot <bench@silentmedia.com>
2022-02-10 17:41 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-02-11 00:12 ` Andres Freund <andres@anarazel.de>
2022-02-11 01:38 ` Michael Paquier <michael@paquier.xyz>
2022-02-11 02:18 ` Andres Freund <andres@anarazel.de>
2022-02-12 17:03 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-03-08 15:44 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-04-20 07:14 ` Michael Paquier <michael@paquier.xyz>
2022-04-20 18:23 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-04-21 03:54 ` Michael Paquier <michael@paquier.xyz>
2022-04-21 16:17 ` Andres Freund <andres@anarazel.de>
2022-04-25 04:47 ` Michael Paquier <michael@paquier.xyz>
2022-04-23 06:02 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-04-24 03:03 ` Noah Misch <noah@leadboat.com>
2022-04-25 05:11 ` Michael Paquier <michael@paquier.xyz>
2022-04-25 05:22 ` Noah Misch <noah@leadboat.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox