pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #18280: logical decoding build wrong snapshot for subtransactions
11+ messages / 6 participants
[nested] [flat]
* BUG #18280: logical decoding build wrong snapshot for subtransactions
@ 2024-01-10 12:51 PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
0 siblings, 1 reply; 11+ messages in thread
From: PG Bug reporting form @ 2024-01-10 12:51 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: feichanghong@qq.com
The following bug has been logged on the website:
Bug reference: 18280
Logged by: Fei Changhong
Email address: feichanghong@qq.com
PostgreSQL version: 15.5
Operating system: Operating system: centos 7,Kernel version: 5.10.13
Description:
Hi, all
I encountered a problem related to logical decoding history snapshot. The
specific phenomenon is "ERROR: could not map filenode "base/5/16390" to
relation OID".
The sub-transaction that modified the catalog ends earlier than the
restart_lsn of the logical replication slot, but the commit wal record of
its parent transaction is after the restart_lsn. The WAL record related to
the sub-transaction will not be decoded during logical decoding, so it will
not be marked as containing catalog changes. The catalog is not recorded in
the committed list of the snapshot.
SnapBuildXidSetCatalogChanges (introduced in 272248a) skipping the check for
the sub-transactions when the parent transaction has been marked as
containing catalog changes should be the root cause of the problem.
The problem can be reproduced by following the following steps (to avoid the
impact of bgwriter writing XLOG_RUNNING_XACTS WAL records, I increased the
value of LOG_SNAPSHOT_INTERVAL_MS):
session 1:
```
CREATE TABLE tbl1 (val1 integer, val2 integer);
CREATE TABLE tbl1_part (val1 integer) PARTITION BY RANGE (val1);
SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot',
'test_decoding');
BEGIN;
SAVEPOINT sp1;
CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO
(10);
RELEASE SAVEPOINT sp1;
```
session 2:
```
CHECKPOINT;
```
session 1:
```
CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO
(20);
COMMIT;
BEGIN;
TRUNCATE tbl1;
```
session 2:
```
CHECKPOINT;
SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
'skip-empty-xacts', '1', 'include-xids', '0');
INSERT INTO tbl1_part VALUES (1);
SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
'skip-empty-xacts', '1', 'include-xids', '0');
```
I will provide a patch to fix this problem later.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re:BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
@ 2024-01-10 13:56 ` =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-12 02:00 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Andy Fan <zhihuifan1213@163.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 11+ messages in thread
From: feichanghong @ 2024-01-10 13:56 UTC (permalink / raw)
To: feichanghong <feichanghong@qq.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
> SnapBuildXidSetCatalogChanges (introduced in 272248a) skipping the check for
> the sub-transactions when the parent transaction has been marked as
> containing catalog changes should be the root cause of the problem.
My fix is to remove the ReorderBufferXidHasCatalogChanges condition from the SnapBuildXidSetCatalogChanges function. This approach has a false positive; the subtransactions that didn't change catalog to the snapshot are also added to the commit list. But that won't be a problem since we use snapshot built during decoding only to read system catalogs, as 272248a's commit message says.
The attached patch has been verified to resolve the mentioned issue.
Best regards.
Alibaba Cloud Computing Ltd.
Attachments:
[application/octet-stream] fix_wrong_snapshot_for_logical_decoding.patch (5.6K, ../../tencent_0726471CA7087C4AC6EF1F19409DF11FCF08@qq.com/3-fix_wrong_snapshot_for_logical_decoding.patch)
download | inline diff:
diff --git a/contrib/test_decoding/expected/catalog_change_snapshot.out b/contrib/test_decoding/expected/catalog_change_snapshot.out
index b33e49c0b1..a2eadc5418 100644
--- a/contrib/test_decoding/expected/catalog_change_snapshot.out
+++ b/contrib/test_decoding/expected/catalog_change_snapshot.out
@@ -132,3 +132,41 @@ COMMIT
stop
(1 row)
+
+starting permutation: s0_init s0_begin s0_savepoint s0_create_part1 s0_savepoint_release s1_checkpoint s0_create_part2 s0_commit s0_begin s0_truncate s1_checkpoint s1_get_changes s0_insert_part s1_get_changes s0_commit
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp1;
+step s0_create_part1: CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO (10);
+step s0_savepoint_release: RELEASE SAVEPOINT sp1;
+step s1_checkpoint: CHECKPOINT;
+step s0_create_part2: CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO (20);
+step s0_commit: COMMIT;
+step s0_begin: BEGIN;
+step s0_truncate: TRUNCATE tbl1;
+step s1_checkpoint: CHECKPOINT;
+step s1_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_part: INSERT INTO tbl1_part VALUES (1);
+step s1_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+--------------------------------------------------
+BEGIN
+table public.tbl1_part_p1: INSERT: val1[integer]:1
+COMMIT
+(3 rows)
+
+step s0_commit: COMMIT;
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/catalog_change_snapshot.spec b/contrib/test_decoding/specs/catalog_change_snapshot.spec
index 770dbd642d..0a17a0dd17 100644
--- a/contrib/test_decoding/specs/catalog_change_snapshot.spec
+++ b/contrib/test_decoding/specs/catalog_change_snapshot.spec
@@ -3,13 +3,16 @@
setup
{
DROP TABLE IF EXISTS tbl1;
+ DROP TABLE IF EXISTS tbl1_part;
CREATE TABLE tbl1 (val1 integer, val2 integer);
+ CREATE TABLE tbl1_part (val1 integer) PARTITION BY RANGE (val1);
CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
}
teardown
{
DROP TABLE tbl1;
+ DROP TABLE tbl1_part;
DROP TABLE user_cat;
SELECT 'stop' FROM pg_drop_replication_slot('isolation_slot');
}
@@ -19,15 +22,19 @@ setup { SET synchronous_commit=on; }
step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); }
step "s0_begin" { BEGIN; }
step "s0_savepoint" { SAVEPOINT sp1; }
+step "s0_savepoint_release" { RELEASE SAVEPOINT sp1; }
step "s0_truncate" { TRUNCATE tbl1; }
step "s0_insert" { INSERT INTO tbl1 VALUES (1); }
step "s0_insert2" { INSERT INTO user_cat VALUES (1); }
+step "s0_create_part1" { CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO (10); }
+step "s0_create_part2" { CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO (20); }
step "s0_commit" { COMMIT; }
session "s1"
setup { SET synchronous_commit=on; }
step "s1_checkpoint" { CHECKPOINT; }
step "s1_get_changes" { SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
# For the transaction that TRUNCATEd the table tbl1, the last decoding decodes
# only its COMMIT record, because it starts from the RUNNING_XACTS record emitted
@@ -60,3 +67,9 @@ permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert" "s1_checkpoint" "s1_
# to skip this xact but ensure that corresponding invalidation messages
# get processed.
permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert" "s1_checkpoint" "s1_get_changes" "s0_truncate" "s0_commit" "s0_begin" "s0_insert" "s1_checkpoint" "s1_get_changes" "s0_commit" "s1_get_changes"
+
+# For the subtransaction that created the table tbl1_part_p1, the last decoding does
+# not decode any WAL records for it, because it ends before the RUNNING_XACTS record.
+# When decoding the COMMIT record of its parent transaction, the subtransaction also
+# needs to be added to the committed list because it modified the catalog.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_create_part1" "s0_savepoint_release" "s1_checkpoint" "s0_create_part2" "s0_commit" "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes" "s0_insert_part" "s1_get_changes" "s0_commit"
\ No newline at end of file
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3eaaaac6b3..91cc70e14b 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -2126,11 +2126,12 @@ SnapBuildXidSetCatalogChanges(SnapBuild *builder, TransactionId xid, int subxcnt
TransactionId *subxacts, XLogRecPtr lsn)
{
/*
- * Skip if there is no initial running xacts information or the
- * transaction is already marked as containing catalog changes.
+ * Skip if there is no initial running xacts information.
+ * Even if the transaction has been marked as containing catalog changes,
+ * it cannot be skipped because its subtransactions that modified the
+ * catalog may not be marked.
*/
- if (NInitialRunningXacts == 0 ||
- ReorderBufferXidHasCatalogChanges(builder->reorder, xid))
+ if (NInitialRunningXacts == 0)
return;
if (bsearch(&xid, InitialRunningXacts, NInitialRunningXacts,
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
@ 2024-01-12 02:00 ` Andy Fan <zhihuifan1213@163.com>
1 sibling, 0 replies; 11+ messages in thread
From: Andy Fan @ 2024-01-12 02:00 UTC (permalink / raw)
To: feichanghong <feichanghong@qq.com>; +Cc: pgsql-bugs@lists.postgresql.org,Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>
Hi,
Thanks for the steps for reproducing this bug and the fix!
"feichanghong" <feichanghong@qq.com> writes:
>> SnapBuildXidSetCatalogChanges (introduced in 272248a) skipping the check for
>> the sub-transactions when the parent transaction has been marked as
>> containing catalog changes should be the root cause of the problem.
>
> My fix is to remove the ReorderBufferXidHasCatalogChanges condition from the SnapBuildXidSetCatalogChanges function. This
> approach has a false positive; the subtransactions that didn't change catalog to the snapshot are also added to the
> commit list. But that won't be a problem since we use snapshot built during decoding only to read system catalogs, as
> 272248a's commit message says.
The patch looks good to me.
Added Masahiko who is the author of commit 272248a and Amit who is alway
good at this area as well. Hope this can get more attention from them.
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
@ 2024-01-23 11:42 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:27 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions feichanghong <feichanghong@qq.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
1 sibling, 2 replies; 11+ messages in thread
From: Amit Kapila @ 2024-01-23 11:42 UTC (permalink / raw)
To: feichanghong <feichanghong@qq.com>; +Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On Wed, Jan 10, 2024 at 7:56 PM feichanghong <feichanghong@qq.com> wrote:
>
> > SnapBuildXidSetCatalogChanges (introduced in 272248a) skipping the check for
> > the sub-transactions when the parent transaction has been marked as
> > containing catalog changes should be the root cause of the problem.
>
> My fix is to remove the ReorderBufferXidHasCatalogChanges condition from the SnapBuildXidSetCatalogChanges function. This approach has a false positive; the subtransactions that didn't change catalog to the snapshot are also added to the commit list. But that won't be a problem since we use snapshot built during decoding only to read system catalogs, as 272248a's commit message says.
>
Thanks, your analysis sounds correct to me. BTW, I think this should
occur only in 15 and 14 because in prior branches we didn't check if
the xact has catalog changes before marking its subxacts. From 16
onwards, we already accurately serialize the xacts and subxacts that
have made any catalog changes, so this problem shouldn't be there. Can
you please once verify the same?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
@ 2024-01-23 12:27 ` feichanghong <feichanghong@qq.com>
1 sibling, 0 replies; 11+ messages in thread
From: feichanghong @ 2024-01-23 12:27 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Thank you for your attention.
> On Jan 23, 2024, at 19:42, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 10, 2024 at 7:56 PM feichanghong <feichanghong@qq.com> wrote:
>>
>>> SnapBuildXidSetCatalogChanges (introduced in 272248a) skipping the check for
>>> the sub-transactions when the parent transaction has been marked as
>>> containing catalog changes should be the root cause of the problem.
>>
>> My fix is to remove the ReorderBufferXidHasCatalogChanges condition from the SnapBuildXidSetCatalogChanges function. This approach has a false positive; the subtransactions that didn't change catalog to the snapshot are also added to the commit list. But that won't be a problem since we use snapshot built during decoding only to read system catalogs, as 272248a's commit message says.
>>
>
> Thanks, your analysis sounds correct to me. BTW, I think this should
> occur only in 15 and 14 because in prior branches we didn't check if
> the xact has catalog changes before marking its subxacts. From 16
> onwards, we already accurately serialize the xacts and subxacts that
> have made any catalog changes, so this problem shouldn't be there. Can
> you please once verify the same?
>
Yes, I verified versions 11 to 16 one by one, and this problem only occurs
in 14 and 15.
Best Regards,
Fei Changhong
Alibaba Cloud Computing Ltd.
^ permalink raw reply [nested|flat] 11+ messages in thread
* RE: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
@ 2024-01-23 12:33 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 11+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-23 12:33 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; feichanghong <feichanghong@qq.com>; +Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Dear hackers,
Thanks for reporting the bug. I could reproduce the issue with your reproducer.
I agreed your point.
> Thanks, your analysis sounds correct to me. BTW, I think this should
> occur only in 15 and 14 because in prior branches we didn't check if
> the xact has catalog changes before marking its subxacts. From 16
> onwards, we already accurately serialize the xacts and subxacts that
> have made any catalog changes, so this problem shouldn't be there. Can
> you please once verify the same?
I tested PG16-PG13 and got below result. As you said, only the ERROR happened
only on PG14 and PG15.
PG16 not happen
PG15 happen
PG14 happen
PG13 not happen
Also, proposed fix could be applied atop REL_14_STABLE and pass the test.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
@ 2024-01-24 09:18 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-24 10:00 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 11:01 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions feichanghong <feichanghong@qq.com>
0 siblings, 2 replies; 11+ messages in thread
From: Amit Kapila @ 2024-01-24 09:18 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: feichanghong <feichanghong@qq.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On Tue, Jan 23, 2024 at 6:03 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Thanks for reporting the bug. I could reproduce the issue with your reproducer.
> I agreed your point.
>
> > Thanks, your analysis sounds correct to me. BTW, I think this should
> > occur only in 15 and 14 because in prior branches we didn't check if
> > the xact has catalog changes before marking its subxacts. From 16
> > onwards, we already accurately serialize the xacts and subxacts that
> > have made any catalog changes, so this problem shouldn't be there. Can
> > you please once verify the same?
>
> I tested PG16-PG13 and got below result. As you said, only the ERROR happened
> only on PG14 and PG15.
>
While looking closely at the test result, I wondered how the following
part of test "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes"
"s0_insert_part" "s1_get_changes" "s0_commit" can see the insert when
the containing transaction is not yet committed. Then, I found it is
because the insert is performed from session 1 due to way it is
declared.
session "s1"
...
step "s1_get_changes" { SELECT data FROM
pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
I think this session should be performed from seesion-1 and we need
one more get_changes() call to see the problem. I have modified the
test accordingly and also changed the comments. See the attached and
let me know what you people think.
--
With Regards,
Amit Kapila.
Attachments:
[application/octet-stream] v2-0001-Fix-catalog-lookup-due-to-wrong-snapshot-for-subt.patch (7.2K, ../../CAA4eK1KULrO8H4nzosuDiV0tAVHpHLuFFcvQA7JB125Qbhc6OQ@mail.gmail.com/2-v2-0001-Fix-catalog-lookup-due-to-wrong-snapshot-for-subt.patch)
download | inline diff:
From 8dde671cfa30c8e85660d42beeb4e51b5d9ca985 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Wed, 24 Jan 2024 13:53:56 +0530
Subject: [PATCH v2] Fix catalog lookup due to wrong snapshot for
subtransactions during decoding.
In commit 272248a0c, we fixed the catalog lookup due to the wrong snapshot
for transactions and subtransactions during decoding. We failed to
consider the case where top-level xact is already marked as containing
catalog change but its subtransaction is not yet marked as containing
catalog change even though it contained such a change.
This can happen when during restart, none of the WAL records from the
subtransaction was decoded and top-level xact contains a DDL.
We fix it by marking the transaction and all its subtransactions as
containing catalog changes if the top-level xact contains any catalog
change and it is present in the initial running xacts array.
This fix is required only for 14 and 15 because in prior branches we
already always mark the transaction and all its subtransactions as
containing catalog changes in the same case. In 16 and above, we preserve
the list of transaction IDs and sub-transaction IDs, that have modified
catalogs and are running during snapshot serialization, to the serialized
snapshot (see commit 7f13ac8123).
Author: Fei Changhong
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/18280-4c8060178cb41750@postgresql.org
---
.../expected/catalog_change_snapshot.out | 44 +++++++++++++++++++
.../specs/catalog_change_snapshot.spec | 15 +++++++
src/backend/replication/logical/snapbuild.c | 10 +++--
3 files changed, 65 insertions(+), 4 deletions(-)
diff --git a/contrib/test_decoding/expected/catalog_change_snapshot.out b/contrib/test_decoding/expected/catalog_change_snapshot.out
index b33e49c0b1..551dc2204a 100644
--- a/contrib/test_decoding/expected/catalog_change_snapshot.out
+++ b/contrib/test_decoding/expected/catalog_change_snapshot.out
@@ -132,3 +132,47 @@ COMMIT
stop
(1 row)
+
+starting permutation: s0_init s0_begin s0_savepoint s0_create_part1 s0_savepoint_release s1_checkpoint s0_create_part2 s0_commit s0_begin s0_truncate s1_checkpoint s1_get_changes s0_insert_part s1_get_changes s0_commit s1_get_changes
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp1;
+step s0_create_part1: CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO (10);
+step s0_savepoint_release: RELEASE SAVEPOINT sp1;
+step s1_checkpoint: CHECKPOINT;
+step s0_create_part2: CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO (20);
+step s0_commit: COMMIT;
+step s0_begin: BEGIN;
+step s0_truncate: TRUNCATE tbl1;
+step s1_checkpoint: CHECKPOINT;
+step s1_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_part: INSERT INTO tbl1_part VALUES (1);
+step s1_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_commit: COMMIT;
+step s1_get_changes: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+--------------------------------------------------
+BEGIN
+table public.tbl1: TRUNCATE: (no-flags)
+table public.tbl1_part_p1: INSERT: val1[integer]:1
+COMMIT
+(4 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/catalog_change_snapshot.spec b/contrib/test_decoding/specs/catalog_change_snapshot.spec
index 770dbd642d..d8b9df97ed 100644
--- a/contrib/test_decoding/specs/catalog_change_snapshot.spec
+++ b/contrib/test_decoding/specs/catalog_change_snapshot.spec
@@ -3,13 +3,16 @@
setup
{
DROP TABLE IF EXISTS tbl1;
+ DROP TABLE IF EXISTS tbl1_part;
CREATE TABLE tbl1 (val1 integer, val2 integer);
+ CREATE TABLE tbl1_part (val1 integer) PARTITION BY RANGE (val1);
CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
}
teardown
{
DROP TABLE tbl1;
+ DROP TABLE tbl1_part;
DROP TABLE user_cat;
SELECT 'stop' FROM pg_drop_replication_slot('isolation_slot');
}
@@ -19,9 +22,13 @@ setup { SET synchronous_commit=on; }
step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding'); }
step "s0_begin" { BEGIN; }
step "s0_savepoint" { SAVEPOINT sp1; }
+step "s0_savepoint_release" { RELEASE SAVEPOINT sp1; }
step "s0_truncate" { TRUNCATE tbl1; }
step "s0_insert" { INSERT INTO tbl1 VALUES (1); }
step "s0_insert2" { INSERT INTO user_cat VALUES (1); }
+step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
+step "s0_create_part1" { CREATE TABLE tbl1_part_p1 PARTITION OF tbl1_part FOR VALUES FROM (0) TO (10); }
+step "s0_create_part2" { CREATE TABLE tbl1_part_p2 PARTITION OF tbl1_part FOR VALUES FROM (10) TO (20); }
step "s0_commit" { COMMIT; }
session "s1"
@@ -60,3 +67,11 @@ permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert" "s1_checkpoint" "s1_
# to skip this xact but ensure that corresponding invalidation messages
# get processed.
permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert" "s1_checkpoint" "s1_get_changes" "s0_truncate" "s0_commit" "s0_begin" "s0_insert" "s1_checkpoint" "s1_get_changes" "s0_commit" "s1_get_changes"
+
+# The last decoding restarts from the first checkpoint and doesn't decode
+# any WAL records generated by the subtransaction that performed s0_create_part1.
+# While processing the commit record for the corresponding top-level transaction
+# which will be marked as containing catalog change even before commit, we ensure
+# that the corresponding substransaction is also marked as containing a catalog
+# modifying change.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_create_part1" "s0_savepoint_release" "s1_checkpoint" "s0_create_part2" "s0_commit" "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes" "s0_insert_part" "s1_get_changes" "s0_commit" "s1_get_changes"
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3eaaaac6b3..39819947c1 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -2126,11 +2126,13 @@ SnapBuildXidSetCatalogChanges(SnapBuild *builder, TransactionId xid, int subxcnt
TransactionId *subxacts, XLogRecPtr lsn)
{
/*
- * Skip if there is no initial running xacts information or the
- * transaction is already marked as containing catalog changes.
+ * Skip if there is no initial running xacts information.
+ *
+ * Even if the transaction has been marked as containing catalog
+ * changes, it cannot be skipped because its subtransactions that
+ * modified the catalog may not be marked.
*/
- if (NInitialRunningXacts == 0 ||
- ReorderBufferXidHasCatalogChanges(builder->reorder, xid))
+ if (NInitialRunningXacts == 0)
return;
if (bsearch(&xid, InitialRunningXacts, NInitialRunningXacts,
--
2.39.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* RE: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
@ 2024-01-24 10:00 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-25 03:58 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 11+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-01-24 10:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: feichanghong <feichanghong@qq.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Dear Amit,
> While looking closely at the test result, I wondered how the following
> part of test "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes"
> "s0_insert_part" "s1_get_changes" "s0_commit" can see the insert when
> the containing transaction is not yet committed. Then, I found it is
> because the insert is performed from session 1 due to way it is
> declared.
>
> session "s1"
> ...
> step "s1_get_changes" { SELECT data FROM
> pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
> 'skip-empty-xacts', '1', 'include-xids', '0'); }
> +step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
>
> I think this session should be performed from seesion-1 and we need
> one more get_changes() call to see the problem. I have modified the
> test accordingly and also changed the comments. See the attached and
> let me know what you people think.
Agreed your point and thanks for updating the patch.
I confirmed your patch passed the test on both PG14 and PG15.
One comment for commit message:
>
This can happen when during restart, none of the WAL records from the
subtransaction was decoded and top-level xact contains a DDL.
>
It may be OK, the word "restart" may be confusing because it does not mean for
the server instance. How about "decode" or "reply"?
If you do not like, it's OK to keep.
Others, LGTM.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-24 10:00 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
@ 2024-01-25 03:58 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-29 09:24 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Amit Kapila @ 2024-01-25 03:58 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: feichanghong <feichanghong@qq.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On Wed, Jan 24, 2024 at 3:30 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> > While looking closely at the test result, I wondered how the following
> > part of test "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes"
> > "s0_insert_part" "s1_get_changes" "s0_commit" can see the insert when
> > the containing transaction is not yet committed. Then, I found it is
> > because the insert is performed from session 1 due to way it is
> > declared.
> >
> > session "s1"
> > ...
> > step "s1_get_changes" { SELECT data FROM
> > pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
> > 'skip-empty-xacts', '1', 'include-xids', '0'); }
> > +step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
> >
> > I think this session should be performed from seesion-1 and we need
> > one more get_changes() call to see the problem. I have modified the
> > test accordingly and also changed the comments. See the attached and
> > let me know what you people think.
>
> Agreed your point and thanks for updating the patch.
> I confirmed your patch passed the test on both PG14 and PG15.
>
> One comment for commit message:
>
> >
> This can happen when during restart, none of the WAL records from the
> subtransaction was decoded and top-level xact contains a DDL.
> >
>
> It may be OK, the word "restart" may be confusing because it does not mean for
> the server instance. How about "decode" or "reply"?
>
Okay, we can change it to 'decoding'. I'll do this before commit
unless there are more comments. I'll push this early next week
(probably on Monday) unless I see any comments.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-24 10:00 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-25 03:58 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
@ 2024-01-29 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: Amit Kapila @ 2024-01-29 09:24 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: feichanghong <feichanghong@qq.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
On Thu, Jan 25, 2024 at 9:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Okay, we can change it to 'decoding'. I'll do this before commit
> unless there are more comments. I'll push this early next week
> (probably on Monday) unless I see any comments.
>
Pushed.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: BUG #18280: logical decoding build wrong snapshot for subtransactions
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` Re:BUG #18280: logical decoding build wrong snapshot for subtransactions =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-23 11:42 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:33 ` RE: BUG #18280: logical decoding build wrong snapshot for subtransactions Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Re: BUG #18280: logical decoding build wrong snapshot for subtransactions Amit Kapila <amit.kapila16@gmail.com>
@ 2024-01-24 11:01 ` feichanghong <feichanghong@qq.com>
1 sibling, 0 replies; 11+ messages in thread
From: feichanghong @ 2024-01-24 11:01 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Thank you for your correction.
> On Jan 24, 2024, at 17:18, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> While looking closely at the test result, I wondered how the following
> part of test "s0_begin" "s0_truncate" "s1_checkpoint" "s1_get_changes"
> "s0_insert_part" "s1_get_changes" "s0_commit" can see the insert when
> the containing transaction is not yet committed. Then, I found it is
> because the insert is performed from session 1 due to way it is
> declared.
>
> session "s1"
> ...
> step "s1_get_changes" { SELECT data FROM
> pg_logical_slot_get_changes('isolation_slot', NULL, NULL,
> 'skip-empty-xacts', '1', 'include-xids', '0'); }
> +step "s0_insert_part" { INSERT INTO tbl1_part VALUES (1); }
>
> I think this session should be performed from seesion-1 and we need
> one more get_changes() call to see the problem. I have modified the
> test accordingly and also changed the comments. See the attached and
> let me know what you people think.
I'm very sorry for confusing you due to my mistake. In my original patch,
"s0_insert_part" should be "s1_insert_part" in session "s1", and after
confirmation, the problem can be reproduced.
Of course your fix is totally correct for me.
Best Regards,
Fei Changhong
Alibaba Cloud Computing Ltd.
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2024-01-29 09:24 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-01-10 12:51 BUG #18280: logical decoding build wrong snapshot for subtransactions PG Bug reporting form <noreply@postgresql.org>
2024-01-10 13:56 ` =?ISO-8859-1?B?ZmVpY2hhbmdob25n?= <feichanghong@qq.com>
2024-01-12 02:00 ` Andy Fan <zhihuifan1213@163.com>
2024-01-23 11:42 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-23 12:27 ` feichanghong <feichanghong@qq.com>
2024-01-23 12:33 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-24 09:18 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-24 10:00 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-01-25 03:58 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-29 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
2024-01-24 11:01 ` feichanghong <feichanghong@qq.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox