From: Michael Paquier <michael@paquier.xyz>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Manuel Rigger <rigger.manuel@gmail.com>
Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: REINDEX CONCURRENTLY unexpectedly fails
Date: Wed, 20 Nov 2019 12:54:08 +0900
Message-ID: <20191120035408.GA4243@paquier.xyz> (raw)
In-Reply-To: <20191120013649.2xfr2dnznyjgluch@alap3.anarazel.de>
References: <12383.1573660748@sss.pgh.pa.us>
<20191113163851.e2enkdpvd3nrobst@alap3.anarazel.de>
<19198.1573663534@sss.pgh.pa.us>
<20191114080855.GG1910@paquier.xyz>
<3808.1573754033@sss.pgh.pa.us>
<20191115024512.GE1849@paquier.xyz>
<20191115025412.p3beiflrd6bx6wgf@alap3.anarazel.de>
<20191115032141.GH1849@paquier.xyz>
<20191115080706.GI1849@paquier.xyz>
<20191120013649.2xfr2dnznyjgluch@alap3.anarazel.de>
On Tue, Nov 19, 2019 at 05:36:49PM -0800, Andres Freund wrote:
> On 2019-11-15 17:07:06 +0900, Michael Paquier wrote:
>> So, here is a patch to address all that. I have also added a test for
>> REINDEX SCHEMA on a temporary schema. While looking at the problem, I
>> have found a crash if trying to reindex concurrently an index or a
>> table using a temporary relation from a different session because we
>> have been lacking checks with RELATION_IS_OTHER_TEMP() in the
>> concurrent code paths.
Thanks for providing comments. The next set of minor releases is in
February, so we have some room until that. For now I'll just add this
patch to the next CF as a bug fix to keeo track of it.
> Probably worth fixing separately?
There is already a check for RELATION_IS_OTHER_TEMP() in the
non-concurrent reindex path, so by forcing temp relations to take this
path then the issue gets fixed automatically, with the error message
from reindex_index() generated.
>> Please note that I have not changed index_drop() for the concurrent
>> mode. Per my checks this does not actually cause any direct bugs as
>> this code path just takes care of dropping the dependencies with the
>> index. There could be an argument for changing that on HEAD though,
>> but I am not sure that this is worth the complication either.
>
> I'm extremely doubtful this works correctly. E.g., what prevents the
> tids for index_concurrently_set_dead() from being recycled and pointing
> to a different row after one of the internal transactions?
ON COMMIT DELETE ROWS does a physical truncation of the relation
files. And as DROP INDEX CONCURRENTLY cannot be run in a transaction
block, you would never face a case where you have no past TIDs which
could be referred to when setting the index as invalid. Now I don't
actually object to enforce the non-concurrent path in index_drop() for
*all* temporary relations. Anyway, that makes sense in itself on
performance grounds, similarly to the create path, so did that by
enforcing the flag in index_drop() (doDeletion would be tempting but I
took the problem at its root). And added some tests for the drop path
and an extra assertion.
> I don't think "ignore concurrent index creation" is a good description -
> they're still created. I'd rephrase it as "For temporary tables build
> index non-concurrently", or something in that vein.
>
> I think we also need to mention somewhere that it's actually crucial to
> ignore them, as otherwise we'd run into problems with on commit truncate
> / drop.
Sure. I have expanded the comment section on that.
> Probably worthwhile to centralize check and comments into one place, to
> avoid duplication / them diverging. Perhaps something like
> IndexCreationSupportsConcurrent() or IndexCreationForceNonConcurrent()?
Good idea, done that. It reduces the explanations of the patch to be
in a single location.
>> @@ -2771,6 +2797,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
>> /* Open relation to get its indexes */
>> heapRelation = table_open(relationOid, ShareUpdateExclusiveLock);
>>
>> + /* Temporary tables are not processed concurrently */
>> + Assert(heapRelation->rd_rel->relpersistence != RELPERSISTENCE_TEMP);
>
> s/are not/can not/?
Okay.
>> +-- Temporary tables with concurrent builds
>> +CREATE TEMP TABLE concur_temp (f1 int, f2 text);
>> +INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');
>> +CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);
>> +DROP TABLE concur_temp;
>> +-- On-commit actions
>> +CREATE TEMP TABLE concur_temp (f1 int, f2 text)
>> + ON COMMIT DELETE ROWS;
>> +INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');
>> +CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);
>> +DROP TABLE concur_temp;
>> --
>
> I'd also add a test for ON COMMIT DROP.
Considered that, but ON COMMIT DROP does not make sense because it
requires a transaction context, which is why I did not add one. And
it seems to me that there is not much value to just check after CIC or
REINDEX's restriction to not run in a transaction block? I added
tests for these two, but I am of the opinion that they don't bring
much.
>> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
>> index 10881ab03a..e5d2b1a06e 100644
>> --- a/doc/src/sgml/ref/reindex.sgml
>> +++ b/doc/src/sgml/ref/reindex.sgml
>> @@ -162,6 +162,9 @@ REINDEX [ ( VERBOSE ) ] { INDEX | TABLE | SCHEMA | DATABASE | SYSTEM } [ CONCURR
>> — see <xref linkend="sql-reindex-concurrently"
>> endterm="sql-reindex-concurrently-title"/>.
>> </para>
>> + <para>
>> + This option is ignored for temporary relations.
>> + </para>
>> </listitem>
>> </varlistentry>
>>
>
> I think either this needs to be more verbose, explaining that there's no
> harm from the option being ignored, or it should be ignored as an
> implementation detail.
I think that documenting it is good for the end-user as well. Just
attempted something in the updated version for all the sections of the
docs involved.
--
Michael
Attachments:reindex-conc-temp-v2.patch (text/x-diff, 14.0K) downloaddiff --git a/src/include/catalog/index.h b/src/include/catalog/index.hindex 1113d25b2d..04d3d4826f 100644--- a/src/include/catalog/index.h+++ b/src/include/catalog/index.h@@ -113,6 +113,8 @@ extern bool CompareIndexInfo(IndexInfo *info1, IndexInfo *info2,
extern void BuildSpeculativeIndexInfo(Relation index, IndexInfo *ii);
+extern bool RelationSupportsConcurrently(char relpersistence);+
extern void FormIndexDatum(IndexInfo *indexInfo,
TupleTableSlot *slot,
EState *estate,
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.cindex 67f637de11..0012ebf69c 100644--- a/src/backend/catalog/index.c+++ b/src/backend/catalog/index.c@@ -2017,6 +2017,13 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
LOCKTAG heaplocktag;
LOCKMODE lockmode;
+ /*+ * Enforce non-concurrent drop if the relation does not support this+ * option.+ */+ if (!RelationSupportsConcurrently(get_rel_persistence(indexId)))+ concurrent = false;+
/*
* To drop an index safely, we must grab exclusive lock on its parent
* table. Exclusive lock on the index alone is insufficient because
@@ -2109,6 +2116,9 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode)
*/
CacheInvalidateRelcache(userHeapRelation);
+ /* Temporary tables cannot be processed concurrently */+ Assert(userHeapRelation->rd_rel->relpersistence != RELPERSISTENCE_TEMP);+
/* save lockrelid and locktag for below, then close but keep locks */
heaprelid = userHeapRelation->rd_lockInfo.lockRelId;
SET_LOCKTAG_RELATION(heaplocktag, heaprelid.dbId, heaprelid.relId);
@@ -2441,6 +2451,30 @@ CompareIndexInfo(IndexInfo *info1, IndexInfo *info2,
return true;
}
+/*+ * RelationSupportsConcurrently+ *+ * Check if a relation supports concurrent builds or not. This is+ * used as a sanity check prior processing CREATE INDEX, DROP INDEX+ * or REINDEX when using CONCURRENTLY.+ */+bool+RelationSupportsConcurrently(char relpersistence)+{+ /*+ * Build indexes non-concurrently for temporary relations. Such+ * relations only work with the session assigned to them, so they are+ * not subject to concurrent concerns, and a concurrent build would+ * cause issues with ON COMMIT actions triggered by the transactions+ * of the concurrent build. A non-concurrent reindex is also more+ * efficient in this case.+ */+ if (relpersistence == RELPERSISTENCE_TEMP)+ return false;++ return true;+}+
/* ----------------
* BuildSpeculativeIndexInfo
* Add extra state to IndexInfo record
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.cindex 374e2d0efe..bb3dcf5525 100644--- a/src/backend/commands/indexcmds.c+++ b/src/backend/commands/indexcmds.c@@ -550,6 +550,13 @@ DefineIndex(Oid relationId,
lockmode = stmt->concurrent ? ShareUpdateExclusiveLock : ShareLock;
rel = table_open(relationId, lockmode);
+ /*+ * Enforce non-concurrent build if the relation does not support this+ * option.+ */+ if (!RelationSupportsConcurrently(rel->rd_rel->relpersistence))+ stmt->concurrent = false;+
namespaceId = RelationGetNamespace(rel);
/* Ensure that it makes sense to index this kind of relation */
@@ -2349,7 +2356,7 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent)
persistence = irel->rd_rel->relpersistence;
index_close(irel, NoLock);
- if (concurrent)+ if (concurrent && RelationSupportsConcurrently(persistence))
ReindexRelationConcurrently(indOid, options);
else
reindex_index(indOid, false, persistence,
@@ -2442,7 +2449,8 @@ ReindexTable(RangeVar *relation, int options, bool concurrent)
0,
RangeVarCallbackOwnsTable, NULL);
- if (concurrent)+ if (concurrent &&+ RelationSupportsConcurrently(get_rel_persistence(heapOid)))
{
result = ReindexRelationConcurrently(heapOid, options);
@@ -2648,7 +2656,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind,
/* functions in indexes may want a snapshot set */
PushActiveSnapshot(GetTransactionSnapshot());
- if (concurrent)+ if (concurrent &&+ RelationSupportsConcurrently(get_rel_persistence(relid)))
{
(void) ReindexRelationConcurrently(relid, options);
/* ReindexRelationConcurrently() does the verbose output */
@@ -2771,6 +2780,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
/* Open relation to get its indexes */
heapRelation = table_open(relationOid, ShareUpdateExclusiveLock);
+ /* Temporary tables cannot be processed concurrently */+ Assert(heapRelation->rd_rel->relpersistence != RELPERSISTENCE_TEMP);+
/* Add all the valid indexes of relation to list */
foreach(lc, RelationGetIndexList(heapRelation))
{
@@ -2864,6 +2876,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
/* Save the list of relation OIDs in private context */
oldcontext = MemoryContextSwitchTo(private_context);
+ /* Temporary tables cannot be processed concurrently */+ Assert(get_rel_persistence(heapId) != RELPERSISTENCE_TEMP);+
/* Track the heap relation of this index for session locks */
heapRelationIds = list_make1_oid(heapId);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.outindex 1cdb7a9663..79cc569969 100644--- a/src/test/regress/expected/create_index.out+++ b/src/test/regress/expected/create_index.out@@ -1398,6 +1398,31 @@ Indexes:
"concur_index5" btree (f2) WHERE f1 = 'x'::text
"std_index" btree (f2)
+-- Temporary tables with concurrent builds and on-commit actions+-- CONCURRENTLY used with CREATE INDEX and DROP INDEX is ignored.+-- PRESERVE ROWS, the default.+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+DROP INDEX CONCURRENTLY concur_temp_ind;+DROP TABLE concur_temp;+-- ON COMMIT DROP+BEGIN;+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT DROP;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+-- Fails when running in a transaction.+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block+COMMIT;+-- ON COMMIT DELETE ROWS+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT DELETE ROWS;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+DROP INDEX CONCURRENTLY concur_temp_ind;+DROP TABLE concur_temp;
--
-- Try some concurrent index drops
--
@@ -2368,6 +2393,55 @@ SELECT pg_get_indexdef('concur_exprs_index_pred_2'::regclass);
(1 row)
DROP TABLE concur_exprs_tab;
+-- Temporary tables and on-commit actions, where CONCURRENTLY is ignored.+-- ON COMMIT PRESERVE ROWS, the default.+CREATE TEMP TABLE concur_temp_tab_1 (c1 int, c2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp_tab_1 VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX concur_temp_ind_1 ON concur_temp_tab_1(c2);+REINDEX TABLE CONCURRENTLY concur_temp_tab_1;+REINDEX INDEX CONCURRENTLY concur_temp_ind_1;+-- Still fails in transaction blocks+BEGIN;+REINDEX INDEX CONCURRENTLY concur_temp_ind_1;+ERROR: REINDEX CONCURRENTLY cannot run inside a transaction block+COMMIT;+-- ON COMMIT DELETE ROWS+CREATE TEMP TABLE concur_temp_tab_2 (c1 int, c2 text)+ ON COMMIT DELETE ROWS;+CREATE INDEX concur_temp_ind_2 ON concur_temp_tab_2(c2);+REINDEX TABLE CONCURRENTLY concur_temp_tab_2;+REINDEX INDEX CONCURRENTLY concur_temp_ind_2;+-- ON COMMIT DROP+BEGIN;+CREATE TEMP TABLE concur_temp_tab_3 (c1 int, c2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp_tab_3 VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX concur_temp_ind_3 ON concur_temp_tab_3(c2);+-- Fails when running in a transaction+REINDEX INDEX CONCURRENTLY concur_temp_ind_3;+ERROR: REINDEX CONCURRENTLY cannot run inside a transaction block+COMMIT;+-- REINDEX SCHEMA processes all temporary relations+CREATE TABLE reindex_temp_before AS+SELECT oid, relname, relfilenode, relkind, reltoastrelid+ FROM pg_class+ WHERE relname IN ('concur_temp_ind_1', 'concur_temp_ind_2');+SELECT pg_my_temp_schema()::regnamespace as temp_schema_name \gset+REINDEX SCHEMA :temp_schema_name;+SELECT b.relname,+ b.relkind,+ CASE WHEN a.relfilenode = b.relfilenode THEN 'relfilenode is unchanged'+ ELSE 'relfilenode has changed' END+ FROM reindex_temp_before b JOIN pg_class a ON b.oid = a.oid+ ORDER BY 1;+ relname | relkind | case +-------------------+---------+-------------------------+ concur_temp_ind_1 | i | relfilenode has changed+ concur_temp_ind_2 | i | relfilenode has changed+(2 rows)++DROP TABLE concur_temp_tab_1, concur_temp_tab_2, reindex_temp_before;
--
-- REINDEX SCHEMA
--
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sqlindex 76598085f7..ae28f08401 100644--- a/src/test/regress/sql/create_index.sql+++ b/src/test/regress/sql/create_index.sql@@ -489,6 +489,31 @@ VACUUM FULL concur_heap;
REINDEX TABLE concur_heap;
\d concur_heap
+-- Temporary tables with concurrent builds and on-commit actions+-- CONCURRENTLY used with CREATE INDEX and DROP INDEX is ignored.+-- PRESERVE ROWS, the default.+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+DROP INDEX CONCURRENTLY concur_temp_ind;+DROP TABLE concur_temp;+-- ON COMMIT DROP+BEGIN;+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT DROP;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+-- Fails when running in a transaction.+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+COMMIT;+-- ON COMMIT DELETE ROWS+CREATE TEMP TABLE concur_temp (f1 int, f2 text)+ ON COMMIT DELETE ROWS;+INSERT INTO concur_temp VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX CONCURRENTLY concur_temp_ind ON concur_temp(f1);+DROP INDEX CONCURRENTLY concur_temp_ind;+DROP TABLE concur_temp;+
--
-- Try some concurrent index drops
--
@@ -954,6 +979,48 @@ SELECT pg_get_indexdef('concur_exprs_index_pred'::regclass);
SELECT pg_get_indexdef('concur_exprs_index_pred_2'::regclass);
DROP TABLE concur_exprs_tab;
+-- Temporary tables and on-commit actions, where CONCURRENTLY is ignored.+-- ON COMMIT PRESERVE ROWS, the default.+CREATE TEMP TABLE concur_temp_tab_1 (c1 int, c2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp_tab_1 VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX concur_temp_ind_1 ON concur_temp_tab_1(c2);+REINDEX TABLE CONCURRENTLY concur_temp_tab_1;+REINDEX INDEX CONCURRENTLY concur_temp_ind_1;+-- Still fails in transaction blocks+BEGIN;+REINDEX INDEX CONCURRENTLY concur_temp_ind_1;+COMMIT;+-- ON COMMIT DELETE ROWS+CREATE TEMP TABLE concur_temp_tab_2 (c1 int, c2 text)+ ON COMMIT DELETE ROWS;+CREATE INDEX concur_temp_ind_2 ON concur_temp_tab_2(c2);+REINDEX TABLE CONCURRENTLY concur_temp_tab_2;+REINDEX INDEX CONCURRENTLY concur_temp_ind_2;+-- ON COMMIT DROP+BEGIN;+CREATE TEMP TABLE concur_temp_tab_3 (c1 int, c2 text)+ ON COMMIT PRESERVE ROWS;+INSERT INTO concur_temp_tab_3 VALUES (1, 'foo'), (2, 'bar');+CREATE INDEX concur_temp_ind_3 ON concur_temp_tab_3(c2);+-- Fails when running in a transaction+REINDEX INDEX CONCURRENTLY concur_temp_ind_3;+COMMIT;+-- REINDEX SCHEMA processes all temporary relations+CREATE TABLE reindex_temp_before AS+SELECT oid, relname, relfilenode, relkind, reltoastrelid+ FROM pg_class+ WHERE relname IN ('concur_temp_ind_1', 'concur_temp_ind_2');+SELECT pg_my_temp_schema()::regnamespace as temp_schema_name \gset+REINDEX SCHEMA :temp_schema_name;+SELECT b.relname,+ b.relkind,+ CASE WHEN a.relfilenode = b.relfilenode THEN 'relfilenode is unchanged'+ ELSE 'relfilenode has changed' END+ FROM reindex_temp_before b JOIN pg_class a ON b.oid = a.oid+ ORDER BY 1;+DROP TABLE concur_temp_tab_1, concur_temp_tab_2, reindex_temp_before;+
--
-- REINDEX SCHEMA
--
diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgmlindex 629a31ef79..e27cc0af3d 100644--- a/doc/src/sgml/ref/create_index.sgml+++ b/doc/src/sgml/ref/create_index.sgml@@ -129,6 +129,11 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ [ IF NOT EXISTS ] <replaceable class=
— see <xref linkend="sql-createindex-concurrently"
endterm="sql-createindex-concurrently-title"/>.
</para>
+ <para>+ This option is ignored for temporary relations as such relations+ are assigned to the session using them, so they are not subject to+ problems with concurrent locking issues.+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/drop_index.sgml b/doc/src/sgml/ref/drop_index.sgmlindex 2a8ca5bf68..c7bd67f24e 100644--- a/doc/src/sgml/ref/drop_index.sgml+++ b/doc/src/sgml/ref/drop_index.sgml@@ -58,6 +58,11 @@ DROP INDEX [ CONCURRENTLY ] [ IF EXISTS ] <replaceable class="parameter">name</r
performed within a transaction block, but
<command>DROP INDEX CONCURRENTLY</command> cannot.
</para>
+ <para>+ This option is ignored for temporary relations as such relations+ are assigned to the session using them, so they are not subject to+ problems with concurrent locking issues.+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgmlindex 10881ab03a..eccdbc66e6 100644--- a/doc/src/sgml/ref/reindex.sgml+++ b/doc/src/sgml/ref/reindex.sgml@@ -162,6 +162,12 @@ REINDEX [ ( VERBOSE ) ] { INDEX | TABLE | SCHEMA | DATABASE | SYSTEM } [ CONCURR
— see <xref linkend="sql-reindex-concurrently"
endterm="sql-reindex-concurrently-title"/>.
</para>
+ <para>+ This option is ignored for temporary relations as such relations+ are assigned to the session using them, so they are not subject to+ concerns with concurrent builds and <command>REINDEX</command>+ requiring an exclusive lock on the parent table.+ </para>
</listitem>
</varlistentry>
signature.asc (application/pgp-signature, 832B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: michael@paquier.xyz, andres@anarazel.de, tgl@sss.pgh.pa.us, rigger.manuel@gmail.com
Subject: Re: REINDEX CONCURRENTLY unexpectedly fails
In-Reply-To: <20191120035408.GA4243@paquier.xyz>
References: <12383.1573660748@sss.pgh.pa.us> <20191113163851.e2enkdpvd3nrobst@alap3.anarazel.de> <19198.1573663534@sss.pgh.pa.us> <20191114080855.GG1910@paquier.xyz> <3808.1573754033@sss.pgh.pa.us> <20191115024512.GE1849@paquier.xyz> <20191115025412.p3beiflrd6bx6wgf@alap3.anarazel.de> <20191115032141.GH1849@paquier.xyz> <20191115080706.GI1849@paquier.xyz> <20191120013649.2xfr2dnznyjgluch@alap3.anarazel.de> <20191120035408.GA4243@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox