From: Noah Misch <noah@leadboat.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: tgl@sss.pgh.pa.us
Cc: pryzby@telsasoft.com
Cc: pgsql-hackers@postgresql.org
Subject: Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation
Date: Tue, 8 Sep 2020 21:05:43 -0700
Message-ID: <20200909040543.GA15541@rfd.leadboat.com> (raw)
In-Reply-To: <20200908024709.GA3640955@rfd.leadboat.com>
References: <20200907.174036.603784943439284546.horikyota.ntt@gmail.com>
<20200907093255.GA3609623@rfd.leadboat.com>
<20200908.091353.1505164218636547516.horikyota.ntt@gmail.com>
<20200908.104332.1790214922230554649.horikyota.ntt@gmail.com>
<20200908024709.GA3640955@rfd.leadboat.com>
On Mon, Sep 07, 2020 at 07:47:09PM -0700, Noah Misch wrote:
> On Tue, Sep 08, 2020 at 10:43:32AM +0900, Kyotaro Horiguchi wrote:
> > At Tue, 08 Sep 2020 09:13:53 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
> > > At Mon, 7 Sep 2020 02:32:55 -0700, Noah Misch <noah@leadboat.com> wrote in
> > > > As a PoC, this looks promising. Thanks. Would you add a test case such that
> > > > the following demonstrates the bug in the absence of your PoC?
> > > >
> > > > printf '%s\n%s\n%s\n' 'log_statement = all' 'wal_level = minimal' 'max_wal_senders = 0' >/tmp/minimal.conf
> > > > make check TEMP_CONFIG=/tmp/minimal.conf
> > >
> > > Mmm. I was close to add some tests to 018_wal_optimize.pl but your
> > > suggestion seems better. I added several ines to create_index.sql.
After looking closer, I've moved the test to reindex_catalog.sql; see that
file's header comment for the reasons. One now needs this command:
printf '%s\n%s\n%s\n' 'log_statement = all' 'wal_level = minimal' 'max_wal_senders = 0' >/tmp/minimal.conf
make check-tests TESTS=reindex_catalog TEMP_CONFIG=/tmp/minimal.conf
> > > > Please have the test try both a nailed-and-mapped relation and a "nailed, but
> > > > not mapped" relation. I am fairly confident that your PoC fixes the former
> > > > case, but the latter may need additional code.
> > >
> > > Mmm. You're right. I choosed pg_amproc_fam_proc_index as
> > > nailed-but-not-mapped index.
> >
> > I fixed a typo (s/staring/starting/).
>
> At a glance, this looks reasonable. If a closer look doesn't reveal problems,
> I'll push this.
RelationBuildDesc() calls RelationInitPhysicalAddr(), so RelationBuildDesc()
can stop calling RelFileNodeSkippingWAL(). The attached version makes it so,
and I plan to push it.
Attachments:skip-wal-nailed-v3.patch (text/plain, 4.4K) download
Author: Noah Misch <noah@leadboat.com>
Commit: Noah Misch <noah@leadboat.com>
Fix rd_firstRelfilenodeSubid for nailed relations, in parallel workers.
Move applicable code out of RelationBuildDesc(), which nailed relations
bypass. Non-assert builds experienced no known problems. Back-patch to
v13, where commit c6b92041d38512a4176ed76ad06f713d2e6c01a8 introduced
rd_firstRelfilenodeSubid.
Kyotaro Horiguchi. Reported by Justin Pryzby.
Discussion: https://postgr.es/m/20200907023737.GA7158@telsasoft.com
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.cindex 96ecad0..9061af8 100644--- a/src/backend/utils/cache/relcache.c+++ b/src/backend/utils/cache/relcache.c@@ -1243,14 +1243,6 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
if (insertIt)
RelationCacheInsert(relation, true);
- /*- * For RelationNeedsWAL() to answer correctly on parallel workers, restore- * rd_firstRelfilenodeSubid. No subtransactions start or end while in- * parallel mode, so the specific SubTransactionId does not matter.- */- if (IsParallelWorker() && RelFileNodeSkippingWAL(relation->rd_node))- relation->rd_firstRelfilenodeSubid = TopSubTransactionId;-
/* It's fully valid */
relation->rd_isvalid = true;
@@ -1273,6 +1265,8 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
static void
RelationInitPhysicalAddr(Relation relation)
{
+ Oid oldnode = relation->rd_node.relNode;+
/* these relations kinds never have storage */
if (!RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
return;
@@ -1330,6 +1324,19 @@ RelationInitPhysicalAddr(Relation relation)
elog(ERROR, "could not find relation mapping for relation \"%s\", OID %u",
RelationGetRelationName(relation), relation->rd_id);
}
++ /*+ * For RelationNeedsWAL() to answer correctly on parallel workers, restore+ * rd_firstRelfilenodeSubid. No subtransactions start or end while in+ * parallel mode, so the specific SubTransactionId does not matter.+ */+ if (IsParallelWorker() && oldnode != relation->rd_node.relNode)+ {+ if (RelFileNodeSkippingWAL(relation->rd_node))+ relation->rd_firstRelfilenodeSubid = TopSubTransactionId;+ else+ relation->rd_firstRelfilenodeSubid = InvalidSubTransactionId;+ }
}
/*
diff --git a/src/test/regress/expected/reindex_catalog.out b/src/test/regress/expected/reindex_catalog.outindex 4b5fba4..204f056 100644--- a/src/test/regress/expected/reindex_catalog.out+++ b/src/test/regress/expected/reindex_catalog.out@@ -36,3 +36,13 @@ REINDEX INDEX pg_index_indexrelid_index; -- non-mapped, non-shared, critical
REINDEX INDEX pg_index_indrelid_index; -- non-mapped, non-shared, non-critical
REINDEX INDEX pg_database_oid_index; -- mapped, shared, critical
REINDEX INDEX pg_shdescription_o_c_index; -- mapped, shared, non-critical
+-- Check the same REINDEX INDEX statements under parallelism.+BEGIN;+SET min_parallel_table_scan_size = 0;+REINDEX INDEX pg_class_oid_index; -- mapped, non-shared, critical+REINDEX INDEX pg_class_relname_nsp_index; -- mapped, non-shared, non-critical+REINDEX INDEX pg_index_indexrelid_index; -- non-mapped, non-shared, critical+REINDEX INDEX pg_index_indrelid_index; -- non-mapped, non-shared, non-critical+REINDEX INDEX pg_database_oid_index; -- mapped, shared, critical+REINDEX INDEX pg_shdescription_o_c_index; -- mapped, shared, non-critical+ROLLBACK;diff --git a/src/test/regress/sql/reindex_catalog.sql b/src/test/regress/sql/reindex_catalog.sqlindex 87ecf52..8203641 100644--- a/src/test/regress/sql/reindex_catalog.sql+++ b/src/test/regress/sql/reindex_catalog.sql@@ -39,3 +39,14 @@ REINDEX INDEX pg_index_indexrelid_index; -- non-mapped, non-shared, critical
REINDEX INDEX pg_index_indrelid_index; -- non-mapped, non-shared, non-critical
REINDEX INDEX pg_database_oid_index; -- mapped, shared, critical
REINDEX INDEX pg_shdescription_o_c_index; -- mapped, shared, non-critical
++-- Check the same REINDEX INDEX statements under parallelism.+BEGIN;+SET min_parallel_table_scan_size = 0;+REINDEX INDEX pg_class_oid_index; -- mapped, non-shared, critical+REINDEX INDEX pg_class_relname_nsp_index; -- mapped, non-shared, non-critical+REINDEX INDEX pg_index_indexrelid_index; -- non-mapped, non-shared, critical+REINDEX INDEX pg_index_indrelid_index; -- non-mapped, non-shared, non-critical+REINDEX INDEX pg_database_oid_index; -- mapped, shared, critical+REINDEX INDEX pg_shdescription_o_c_index; -- mapped, shared, non-critical+ROLLBACK;
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: noah@leadboat.com, horikyota.ntt@gmail.com, tgl@sss.pgh.pa.us, pryzby@telsasoft.com
Subject: Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation
In-Reply-To: <20200909040543.GA15541@rfd.leadboat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox