pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedPotential data loss due to race condition during logical replication slot creation
40+ messages / 6 participants
[nested] [flat]
* Potential data loss due to race condition during logical replication slot creation
@ 2024-02-01 22:28 Callahan, Drew <callaan@amazon.com>
0 siblings, 1 reply; 40+ messages in thread
From: Callahan, Drew @ 2024-02-01 22:28 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; +Cc: sawada.mshk@gmail.com <sawada.mshk@gmail.com>
Hello,
We discovered a race condition during logical replication slot creation that can result in the changes for transactions running at the time of the slot creation to only be partially replicated. We found the cause was due to the slot transitioning from an inconsistent or partially consistent state to a fully consistent state when restoring a snapshot that had been persisted to disk by a different logical slot. We provide a simple reproduction of this issue below:
Session 1:
SELECT pg_create_logical_replication_slot('slot1', 'test_decoding');
CREATE TABLE test (a int);
BEGIN;
INSERT INTO test VALUES (1);
Session 2:
SELECT pg_create_logical_replication_slot('slot2', 'test_decoding');
<query hangs>
Session 3:
CHECKPOINT;
select pg_logical_slot_get_changes('slot1', NULL, NULL);
<should return nothing of interest>
Session 1:
INSERT INTO test VALUES (2);
COMMIT;
<Session 2 query no longer hangs and successfully creates the slot2>
Session 2:
select pg_logical_slot_get_changes('slot1', NULL, NULL);
select pg_logical_slot_get_changes('slot2', NULL, NULL);
<expected: no rows of the txn are returned for slot2>
<actual: The 2nd row of the txn is returned for slot2>
Newly created logical replication slots initialize their restart LSN to the current insert position within the WAL and also force a checkpoint to get the current state of the running transactions on the system. This create process will then wait for all of the transactions within that running xact record to complete before being able to transition to the next snapbuild state. During this time period, if another running xact record is written and then a different logical replication process decodes this running xact record, a globally accessible snapshot will be persisted to disk.
Once all of the transactions from the initial running xact have finished, the process performing the slot creation will become unblocked and will then consume the new running xact record. The process will see a valid snapshot, restore that snapshot from disk, and then transition immediately to the consistent state. The slot will then set the confirmed flush LSN of the slot to the start of the next record after that running xact.
We now have a logical replication slot that has a restart LSN after the changes of transactions that will commit after our confirmed flushed LSN in any case where the two running xact records are not fully disjointed. Once those transactions commit, we will then partially stream their changes.
The attached fix addresses the issue by providing the snapshot builder with enough context to understand whether it is building a snapshot for a slot for the first time or if this is a previously existing slot. This is similar to the “need_full_snapshot” parameter which is already set by the caller to control when and how the snapbuilder is allowed to become consistent.
With this context, the snapshot builder can always skip performing snapshot restore in order to become fully consistent. Since this issue only occurs when the the logical replication slot consumes a persisted snapshot to become fully consistent we can prevent the issue by disallowing this behavior.
Thanks,
Drew
Attachments:
[application/octet-stream] skip_snap_restore.patch (3.3K, ../../29273AF3-9684-4069-9257-D05090B03B99@amazon.com/3-skip_snap_restore.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187a..4e4e0ff42cb 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -441,6 +441,8 @@ CreateInitDecodingContext(const char *plugin,
xl_routine, prepare_write, do_write,
update_progress);
+ SetSnapBuildCreateMode(ctx->context);
+
/* call output plugin initialization callback */
old_context = MemoryContextSwitchTo(ctx->context);
if (ctx->callbacks.startup_cb != NULL)
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e16..372f1934677 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -278,6 +278,15 @@ struct SnapBuild
static ResourceOwner SavedResourceOwnerDuringExport = NULL;
static bool ExportInProgress = false;
+/*
+ * Used to mark the current snapbuild instance in a create mode.
+ * This create mode is required to prevent the snapbuilder from
+ * restoring a snapshot from disk to become consistent as this can
+ * cause the restart and confirmed flush LSNs of the slot to become
+ * initialized to unsafe values.
+ */
+static bool InCreate = false;
+
/* ->committed and ->catchange manipulation */
static void SnapBuildPurgeOlderTxn(SnapBuild *builder);
@@ -443,6 +452,25 @@ SnapBuildSnapIncRefcount(Snapshot snap)
snap->active_count++;
}
+/* Memory context reset callback for clearing the create mode flag */
+static void
+SnapBuildResetCreateModeCallback(void *arg)
+{
+ InCreate = false;
+}
+
+void
+SetSnapBuildCreateMode(MemoryContext context)
+{
+ MemoryContextCallback *mcallback;
+
+ mcallback = palloc0(sizeof(MemoryContextCallback));
+ mcallback->func = SnapBuildResetCreateModeCallback;
+ MemoryContextRegisterResetCallback(context, mcallback);
+
+ InCreate = true;
+}
+
/*
* Decrease refcount of a snapshot and free if the refcount reaches zero.
*
@@ -1391,8 +1419,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /* b) valid on disk state, we are an exisiting slot, and not building full snapshot */
else if (!builder->building_full_snapshot &&
+ !InCreate &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index f49b941b53e..3aff45dced1 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -64,6 +64,7 @@ extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
bool need_full_snapshot,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
+extern void SetSnapBuildCreateMode(MemoryContext context);
extern void SnapBuildSnapDecRefcount(Snapshot snap);
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-02 00:53 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Callahan, Drew <callaan@amazon.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-02-02 00:53 UTC (permalink / raw)
To: Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hi,
Thank you for the report!
On Fri, Feb 2, 2024 at 7:28 AM Callahan, Drew <callaan@amazon.com> wrote:
>
> Hello,
>
> We discovered a race condition during logical replication slot creation that can result in the changes for transactions running at the time of the slot creation to only be partially replicated. We found the cause was due to the slot transitioning from an inconsistent or partially consistent state to a fully consistent state when restoring a snapshot that had been persisted to disk by a different logical slot. We provide a simple reproduction of this issue below:
>
> Session 1:
>
> SELECT pg_create_logical_replication_slot('slot1', 'test_decoding');
>
> CREATE TABLE test (a int);
> BEGIN;
> INSERT INTO test VALUES (1);
>
> Session 2:
>
> SELECT pg_create_logical_replication_slot('slot2', 'test_decoding');
>
> <query hangs>
>
> Session 3:
>
> CHECKPOINT;
>
> select pg_logical_slot_get_changes('slot1', NULL, NULL);
>
> <should return nothing of interest>
>
> Session 1:
>
> INSERT INTO test VALUES (2);
> COMMIT;
>
> <Session 2 query no longer hangs and successfully creates the slot2>
>
> Session 2:
>
> select pg_logical_slot_get_changes('slot1', NULL, NULL);
>
> select pg_logical_slot_get_changes('slot2', NULL, NULL);
>
> <expected: no rows of the txn are returned for slot2>
> <actual: The 2nd row of the txn is returned for slot2>
>
I've confirmed this happens HEAD and all supported backbranches. And I
think it's a (long-standing) bug.
I think we can prepare an isolation test of the above steps and
include it in contrib/test_decoding/specs.
> Newly created logical replication slots initialize their restart LSN to the current insert position within the WAL and also force a checkpoint to get the current state of the running transactions on the system. This create process will then wait for all of the transactions within that running xact record to complete before being able to transition to the next snapbuild state. During this time period, if another running xact record is written and then a different logical replication process decodes this running xact record, a globally accessible snapshot will be persisted to disk.
>
> Once all of the transactions from the initial running xact have finished, the process performing the slot creation will become unblocked and will then consume the new running xact record. The process will see a valid snapshot, restore that snapshot from disk, and then transition immediately to the consistent state. The slot will then set the confirmed flush LSN of the slot to the start of the next record after that running xact.
My analysis is the same. After the slot creation, the slot's LSNs become like:
(tx-begin) < (tx's some changes) < slot's resetart_lsn < (fx's further
some changes) < slot's confirmed_flush_lsn < (tx-commit)
> We now have a logical replication slot that has a restart LSN after the changes of transactions that will commit after our confirmed flushed LSN in any case where the two running xact records are not fully disjointed. Once those transactions commit, we will then partially stream their changes.
>
> The attached fix addresses the issue by providing the snapshot builder with enough context to understand whether it is building a snapshot for a slot for the first time or if this is a previously existing slot. This is similar to the “need_full_snapshot” parameter which is already set by the caller to control when and how the snapbuilder is allowed to become consistent.
>
> With this context, the snapshot builder can always skip performing snapshot restore in order to become fully consistent. Since this issue only occurs when the the logical replication slot consumes a persisted snapshot to become fully consistent we can prevent the issue by disallowing this behavior.
I basically agree with the basic idea of skipping snapshot restore
while finding the initial start point. But I'd like to hear other
opinions too.
Another simple way to fix this is that we always set
need_full_snapshot to true when slot creation. But it would then make
the snapshot builder include non-catalog-change transactions too while
finding the initial startpoint. Which is not necessary.
As for the proposed patch, it uses a global variable, InCreate, and
memory context callback to reset it. While I believe this is for not
breaking any ABI as this bug exists on back branches too, I think it's
better to consider how to fix it in HEAD and then how to backpatch it.
We don't necessarily need to have the same fixes for all branches all
the time.
IIUC we need to skip snapshot restore only when we're finding the
initial start point. So probably we don't need to set InCreate when
calling create_logical_replication_slot() with find_startpoint =
false.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-03 03:12 Callahan, Drew <callaan@amazon.com>
0 siblings, 2 replies; 40+ messages in thread
From: Callahan, Drew @ 2024-02-03 03:12 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>
> I think we can prepare an isolation test of the above steps and
> include it in contrib/test_decoding/specs.
Thanks for the suggestion. Included an isolation test for the
exact repro and a couple other tests for fully before and fully after.
These two tests don't add a whole lot of value since this
is already heavily tested elsewhere, so I'm good with removing.
However, thought it made sense to include from a perspective of
totality given the test name.
> Another simple way to fix this is that we always set
> need_full_snapshot to true when slot creation. But it would then make
> the snapshot builder include non-catalog-change transactions too while
> finding the initial startpoint. Which is not necessary.
Since we're not accumulating changes anyway and we never distribute snapshots
while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
This seems pretty contained and wouldn't require the use of a global variable
and memory context callbacks which is a bit nicer from a readability standpoint.
It's a little inefficient with memory and CPU, especially in the presence of
long running transactions, but likely would not be noticeable to most users.
> As for the proposed patch, it uses a global variable, InCreate, and
> memory context callback to reset it. While I believe this is for not
> breaking any ABI as this bug exists on back branches too, I think it's
> better to consider how to fix it in HEAD and then how to backpatch it.
> We don't necessarily need to have the same fixes for all branches all
> the time.
Attached a patch that is ABI breaking, but removes the need for a global variable.
This relies on pushing the context of whether we are in a create mode to the
logical decoding context, which is cleaner, but not as clean as just pushing
all the way down to the snapbuilder. I would push down to the snap builder,
but the entire object is persisted to disk. This is problematic to me since the
create flag would be persisted causing exisiting slots to restore a snapbuild
that could have the create flag set to true. This is not problematic as is,
since the variable would only be leveraged in SnapBuildRestore() which is only
called while inconsistent, but seems like a future bug waiting to happen.
We could always clear the flag when persisting, but that seems a bit hacky.
We could also fork the snap builder into on-disk and in-memory structs, which
would be convenient since we would not need to invalidate exisiting snapshots.
However, not sure how people feel about the fork.
> IIUC we need to skip snapshot restore only when we're finding the
> initial start point. So probably we don't need to set InCreate when
> calling create_logical_replication_slot() with find_startpoint =
> false.
Yep, that's correct. We can safely skip setting InCreate. We could move
the setter function slotfuncs.c as one way of doing this. I lean towards
setting it no matter what since it is harmless if we never try to find
a decoding point and removes the need for future callers of
CreateInitDecodingContext() from having to determine if it's safe to be false.
Though I agree that InCreate may a bigger hammer than needed which may need
to be modified in the future.
Thanks for the quick response,
Drew Callahan
Attachments:
[application/octet-stream] isolation_test.patch (5.3K, ../../2444AA15-D21B-4CCE-8052-52C7C2DAFE5C@amazon.com/2-isolation_test.patch)
download | inline diff:
diff --git a/contrib/test_decoding/expected/slot_creation_consistency.out b/contrib/test_decoding/expected/slot_creation_consistency.out
new file mode 100755
index 0000000000..4ee9ffea5d
--- /dev/null
+++ b/contrib/test_decoding/expected/slot_creation_consistency.out
@@ -0,0 +1,133 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s2_create s1_b s1_c s3_slot1_get s3_slot2_get
+s1: NOTICE: table "test" does not exist, skipping
+?column?
+--------
+init
+(1 row)
+
+step s2_create: SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s1_b:
+BEGIN;
+INSERT INTO test VALUES (1);
+
+step s1_c:
+INSERT INTO test VALUES (2);
+COMMIT;
+
+step s3_slot1_get: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+---------------------------------------
+BEGIN
+table public.test: INSERT: a[integer]:1
+table public.test: INSERT: a[integer]:2
+COMMIT
+(4 rows)
+
+step s3_slot2_get: SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+---------------------------------------
+BEGIN
+table public.test: INSERT: a[integer]:1
+table public.test: INSERT: a[integer]:2
+COMMIT
+(4 rows)
+
+pg_drop_replication_slot
+------------------------
+
+(1 row)
+
+
+starting permutation: s1_b s2_create s3_checkpoint s3_slot1_get s1_c s3_slot1_get s3_slot2_get
+?column?
+--------
+init
+(1 row)
+
+step s1_b:
+BEGIN;
+INSERT INTO test VALUES (1);
+
+step s2_create: SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding'); <waiting ...>
+step s3_checkpoint: CHECKPOINT;
+step s3_slot1_get: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+----
+(0 rows)
+
+step s1_c:
+INSERT INTO test VALUES (2);
+COMMIT;
+
+step s2_create: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s3_slot1_get: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+---------------------------------------
+BEGIN
+table public.test: INSERT: a[integer]:1
+table public.test: INSERT: a[integer]:2
+COMMIT
+(4 rows)
+
+step s3_slot2_get: SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+----
+(0 rows)
+
+pg_drop_replication_slot
+------------------------
+
+(1 row)
+
+
+starting permutation: s1_b s1_c s2_create s3_slot1_get s3_slot2_get
+?column?
+--------
+init
+(1 row)
+
+step s1_b:
+BEGIN;
+INSERT INTO test VALUES (1);
+
+step s1_c:
+INSERT INTO test VALUES (2);
+COMMIT;
+
+step s2_create: SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s3_slot1_get: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+---------------------------------------
+BEGIN
+table public.test: INSERT: a[integer]:1
+table public.test: INSERT: a[integer]:2
+COMMIT
+(4 rows)
+
+step s3_slot2_get: SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+data
+----
+(0 rows)
+
+pg_drop_replication_slot
+------------------------
+
+(1 row)
+
diff --git a/contrib/test_decoding/specs/slot_creation_consistency.spec b/contrib/test_decoding/specs/slot_creation_consistency.spec
new file mode 100755
index 0000000000..e988872e65
--- /dev/null
+++ b/contrib/test_decoding/specs/slot_creation_consistency.spec
@@ -0,0 +1,48 @@
+# Test that a slot will correctly replicate changes only from transactions that
+# started after the slot was created. This is primarily done to validate that a slot
+# cannot become consistent prematurely through a snapshot that was persisted by
+# a different slot leading to a transaction being partially replicated.
+
+session "s1"
+setup {
+SET synchronous_commit=on;
+BEGIN;
+DROP TABLE IF EXISTS test;
+CREATE TABLE test (a INT);
+COMMIT;
+SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding');
+}
+
+step s1_b {
+BEGIN;
+INSERT INTO test VALUES (1);
+}
+step s1_c {
+INSERT INTO test VALUES (2);
+COMMIT;
+}
+
+teardown {
+SELECT pg_drop_replication_slot('slot1');
+SELECT pg_drop_replication_slot('slot2');
+}
+
+session s2
+setup { SET synchronous_commit=on; }
+step s2_create { SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding'); }
+
+session s3
+setup { SET synchronous_commit=on; }
+step s3_checkpoint { CHECKPOINT; }
+step s3_slot1_get { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); }
+step s3_slot2_get { SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); }
+
+# Create and slot and then make changes. Already done heavily elsewhere, but addded for totality
+permutation s2_create s1_b s1_c s3_slot1_get s3_slot2_get
+
+# Create a slot in the presence of an active transaction. The checkpoint and slot
+# consumption will persist a snapshot to disk ahead of the current read position of the slot.
+permutation s1_b s2_create s3_checkpoint s3_slot1_get s1_c s3_slot1_get s3_slot2_get
+
+# Create a slot after the completion of the active transaction
+permutation s1_b s1_c s2_create s3_slot1_get s3_slot2_get
[application/octet-stream] ctx_skip_snap_restore.patch (6.0K, ../../2444AA15-D21B-4CCE-8052-52C7C2DAFE5C@amazon.com/3-ctx_skip_snap_restore.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index d91055a440..285cc30298 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -371,7 +371,7 @@ standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
xl_running_xacts *running = (xl_running_xacts *) XLogRecGetData(r);
- SnapBuildProcessRunningXacts(builder, buf->origptr, running);
+ SnapBuildProcessRunningXacts(builder, buf->origptr, ctx->in_create, running);
/*
* Abort all transactions that we keep track of, that are
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..77400dc2d4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -151,6 +151,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -200,6 +201,7 @@ StartupDecodingContext(List *output_plugin_options,
}
ctx->slot = slot;
+ ctx->in_create = in_create;
ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, ctx);
if (!ctx->reader)
@@ -437,9 +439,9 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
- xl_routine, prepare_write, do_write,
- update_progress);
+ need_full_snapshot, false /* fast_forward */,
+ true /* in_create */, xl_routine,
+ prepare_write, do_write, update_progress);
/* call output plugin initialization callback */
old_context = MemoryContextSwitchTo(ctx->context);
@@ -572,9 +574,10 @@ CreateDecodingContext(XLogRecPtr start_lsn,
}
ctx = StartupDecodingContext(output_plugin_options,
- start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
- do_write, update_progress);
+ start_lsn, InvalidTransactionId,
+ false /* need_full_snapshot */, fast_forward,
+ false /* in_create */, xl_routine,
+ prepare_write, do_write, update_progress);
/* call output plugin initialization callback */
old_context = MemoryContextSwitchTo(ctx->context);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..f846710843 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -294,7 +294,8 @@ static inline bool SnapBuildXidHasCatalogChanges(SnapBuild *builder, Transaction
uint32 xinfo);
/* xlog reading helper functions for SnapBuildProcessRunningXacts */
-static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running);
+static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn,
+ const bool in_create, xl_running_xacts *running);
static void SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff);
/* serialization functions */
@@ -1210,7 +1211,8 @@ SnapBuildXidHasCatalogChanges(SnapBuild *builder, TransactionId xid,
* anymore.
*/
void
-SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
+SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn,
+ const bool in_create, xl_running_xacts *running)
{
ReorderBufferTXN *txn;
TransactionId xmin;
@@ -1223,7 +1225,7 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
if (builder->state < SNAPBUILD_CONSISTENT)
{
/* returns false if there's no point in performing cleanup just yet */
- if (!SnapBuildFindSnapshot(builder, lsn, running))
+ if (!SnapBuildFindSnapshot(builder, lsn, in_create, running))
return;
}
else
@@ -1312,7 +1314,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
* using the xl_running_xacts record.
*/
static bool
-SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
+SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, const bool in_create,
+ xl_running_xacts *running)
{
/* ---
* Build catalog decoding snapshot incrementally using information about
@@ -1391,8 +1394,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /* b) valid on disk state, we are an exisiting slot, and not building full snapshot */
else if (!builder->building_full_snapshot &&
+ !in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 5f49554ea0..39a27f3db7 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -50,6 +50,12 @@ typedef struct LogicalDecodingContext
*/
bool fast_forward;
+ /*
+ * Marks the logical decoding context as being used for the initial creation
+ * of a logical replication slot.
+ */
+ bool in_create;
+
OutputPluginCallbacks callbacks;
OutputPluginOptions options;
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index f49b941b53..b84403f511 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -88,7 +88,7 @@ extern void SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid,
XLogRecPtr lsn,
struct xl_heap_new_cid *xlrec);
extern void SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn,
- struct xl_running_xacts *running);
+ const bool in_create, struct xl_running_xacts *running);
extern void SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn);
#endif /* SNAPBUILD_H */
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-03 13:48 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: vignesh C @ 2024-02-03 13:48 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, 2 Feb 2024 at 06:24, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Hi,
>
> Thank you for the report!
>
> On Fri, Feb 2, 2024 at 7:28 AM Callahan, Drew <callaan@amazon.com> wrote:
> >
> > Hello,
> >
> > We discovered a race condition during logical replication slot creation that can result in the changes for transactions running at the time of the slot creation to only be partially replicated. We found the cause was due to the slot transitioning from an inconsistent or partially consistent state to a fully consistent state when restoring a snapshot that had been persisted to disk by a different logical slot. We provide a simple reproduction of this issue below:
> >
> > Session 1:
> >
> > SELECT pg_create_logical_replication_slot('slot1', 'test_decoding');
> >
> > CREATE TABLE test (a int);
> > BEGIN;
> > INSERT INTO test VALUES (1);
> >
> > Session 2:
> >
> > SELECT pg_create_logical_replication_slot('slot2', 'test_decoding');
> >
> > <query hangs>
> >
> > Session 3:
> >
> > CHECKPOINT;
> >
> > select pg_logical_slot_get_changes('slot1', NULL, NULL);
> >
> > <should return nothing of interest>
> >
> > Session 1:
> >
> > INSERT INTO test VALUES (2);
> > COMMIT;
> >
> > <Session 2 query no longer hangs and successfully creates the slot2>
> >
> > Session 2:
> >
> > select pg_logical_slot_get_changes('slot1', NULL, NULL);
> >
> > select pg_logical_slot_get_changes('slot2', NULL, NULL);
> >
> > <expected: no rows of the txn are returned for slot2>
> > <actual: The 2nd row of the txn is returned for slot2>
> >
>
> I've confirmed this happens HEAD and all supported backbranches. And I
> think it's a (long-standing) bug.
>
> I think we can prepare an isolation test of the above steps and
> include it in contrib/test_decoding/specs.
>
> > Newly created logical replication slots initialize their restart LSN to the current insert position within the WAL and also force a checkpoint to get the current state of the running transactions on the system. This create process will then wait for all of the transactions within that running xact record to complete before being able to transition to the next snapbuild state. During this time period, if another running xact record is written and then a different logical replication process decodes this running xact record, a globally accessible snapshot will be persisted to disk.
> >
> > Once all of the transactions from the initial running xact have finished, the process performing the slot creation will become unblocked and will then consume the new running xact record. The process will see a valid snapshot, restore that snapshot from disk, and then transition immediately to the consistent state. The slot will then set the confirmed flush LSN of the slot to the start of the next record after that running xact.
>
> My analysis is the same. After the slot creation, the slot's LSNs become like:
>
> (tx-begin) < (tx's some changes) < slot's resetart_lsn < (fx's further
> some changes) < slot's confirmed_flush_lsn < (tx-commit)
>
> > We now have a logical replication slot that has a restart LSN after the changes of transactions that will commit after our confirmed flushed LSN in any case where the two running xact records are not fully disjointed. Once those transactions commit, we will then partially stream their changes.
> >
> > The attached fix addresses the issue by providing the snapshot builder with enough context to understand whether it is building a snapshot for a slot for the first time or if this is a previously existing slot. This is similar to the “need_full_snapshot” parameter which is already set by the caller to control when and how the snapbuilder is allowed to become consistent.
> >
> > With this context, the snapshot builder can always skip performing snapshot restore in order to become fully consistent. Since this issue only occurs when the the logical replication slot consumes a persisted snapshot to become fully consistent we can prevent the issue by disallowing this behavior.
>
> I basically agree with the basic idea of skipping snapshot restore
> while finding the initial start point. But I'd like to hear other
> opinions too.
>
> Another simple way to fix this is that we always set
> need_full_snapshot to true when slot creation. But it would then make
> the snapshot builder include non-catalog-change transactions too while
> finding the initial startpoint. Which is not necessary.
>
> As for the proposed patch, it uses a global variable, InCreate, and
> memory context callback to reset it. While I believe this is for not
> breaking any ABI as this bug exists on back branches too, I think it's
> better to consider how to fix it in HEAD and then how to backpatch it.
> We don't necessarily need to have the same fixes for all branches all
> the time.
I agree to having a separate patch for HEAD in this case. How about a
fix like a patch attached.
Regards,
Vignesh
Attachments:
[text/x-patch] v2-0001-While-a-new-logical-replication-slot-is-created-a.patch (8.8K, ../../CALDaNm02GHrYQ1rMaEzH9_-U0XWoTP_bJpV5XYq9WBmPRE1kAQ@mail.gmail.com/2-v2-0001-While-a-new-logical-replication-slot-is-created-a.patch)
download | inline diff:
From 0b4a7662daa3876f5fbaba277389907fd2ec65bb Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Sat, 3 Feb 2024 19:09:02 +0530
Subject: [PATCH v2] While a new logical replication slot is created, an
already persisted snapshot was being used.
While creating a new logical replication slot, skip restoring a persisted
snapshot to prevent data inconsistency.
---
contrib/test_decoding/Makefile | 3 +-
.../slot_create_skip_exist_snapshot.out | 52 +++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../slot_create_skip_exist_snapshot.spec | 34 ++++++++++++
src/backend/replication/logical/logical.c | 2 +
src/backend/replication/logical/snapbuild.c | 22 +++++++-
src/include/replication/snapbuild.h | 1 +
7 files changed, 113 insertions(+), 2 deletions(-)
create mode 100644 contrib/test_decoding/expected/slot_create_skip_exist_snapshot.out
create mode 100644 contrib/test_decoding/specs/slot_create_skip_exist_snapshot.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..dacc383898 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ slot_create_skip_exist_snapshot
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/slot_create_skip_exist_snapshot.out b/contrib/test_decoding/expected/slot_create_skip_exist_snapshot.out
new file mode 100644
index 0000000000..eb5a26d0ac
--- /dev/null
+++ b/contrib/test_decoding/expected/slot_create_skip_exist_snapshot.out
@@ -0,0 +1,52 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes s0_insert2 s0_commit s1_get_changes_slot1 s1_get_changes_slot2 s1_drop_slot s0_drop_slot
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl1 VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl1 VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0');
+data
+------------------------------------------
+BEGIN
+table public.tbl1: INSERT: val1[integer]:1
+table public.tbl1: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot2: SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s1_drop_slot: SELECT pg_drop_replication_slot('slot2');
+pg_drop_replication_slot
+------------------------
+
+(1 row)
+
+step s0_drop_slot: SELECT pg_drop_replication_slot('slot1');
+pg_drop_replication_slot
+------------------------
+
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..b665039eb7 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'slot_create_skip_exist_snapshot',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/slot_create_skip_exist_snapshot.spec b/contrib/test_decoding/specs/slot_create_skip_exist_snapshot.spec
new file mode 100644
index 0000000000..1c15d12647
--- /dev/null
+++ b/contrib/test_decoding/specs/slot_create_skip_exist_snapshot.spec
@@ -0,0 +1,34 @@
+# When a new slot is created, it should not use the existing snapshots available.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl1;
+ CREATE TABLE tbl1 (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl1;
+}
+
+session "s0"
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl1 VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl1 VALUES (2); }
+step "s0_commit" { COMMIT; }
+step "s0_drop_slot" { SELECT pg_drop_replication_slot('slot1'); }
+
+session "s1"
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0'); }
+step "s1_get_changes_slot2" { SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL, 'include-xids', '0'); }
+step "s1_drop_slot" { SELECT pg_drop_replication_slot('slot2'); }
+
+session "s2"
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'include-xids', '0'); }
+
+# During s2_checkpoint, slot1's snapshot will be persisted. When slot2
+# is creating a snapshot any or the existing snapshots should not be used.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes" "s0_insert2" "s0_commit" "s1_get_changes_slot1" "s1_get_changes_slot2" "s1_drop_slot" "s0_drop_slot"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index ca09c683f1..f4e9a5312a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -633,6 +633,8 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
{
ReplicationSlot *slot = ctx->slot;
+ SnapBuildSkipReadPeristedSnapshot(ctx->snapshot_builder);
+
/* Initialize from where to start reading WAL. */
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index a0b7947d2f..419d460f9c 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -212,6 +212,15 @@ struct SnapBuild
*/
TransactionId next_phase_at;
+ /*
+ * Used to mark the current snapbuild instance in a create mode.
+ * This create mode is required to prevent the snapbuilder from
+ * restoring a snapshot from disk to become consistent as this can
+ * cause the restart and confirmed flush LSNs of the slot to become
+ * initialized to unsafe values.
+ */
+ bool skip_read_persisted_snapshot;
+
/*
* Array of transactions which could have catalog changes that committed
* between xmin and xmax.
@@ -350,6 +359,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->start_decoding_at = start_lsn;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
+ builder->skip_read_persisted_snapshot = false;
MemoryContextSwitchTo(oldcontext);
@@ -375,6 +385,15 @@ FreeSnapshotBuilder(SnapBuild *builder)
MemoryContextDelete(context);
}
+/*
+ * Skip reading a persisted snapshot.
+ */
+void
+SnapBuildSkipReadPeristedSnapshot(SnapBuild *builder)
+{
+ builder->skip_read_persisted_snapshot = true;
+}
+
/*
* Free an unreferenced snapshot that has previously been built by us.
*/
@@ -1396,8 +1415,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /* b) valid on disk state, we are an exisiting slot, and not building full snapshot */
else if (!builder->building_full_snapshot &&
+ !builder->skip_read_persisted_snapshot &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index fbdf362396..f14d7b8f7a 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -65,6 +65,7 @@ extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
+extern void SnapBuildSkipReadPeristedSnapshot(SnapBuild *builder);
extern void SnapBuildSnapDecRefcount(Snapshot snap);
extern Snapshot SnapBuildInitialSnapshot(SnapBuild *builder);
--
2.34.1
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-05 02:06 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Callahan, Drew <callaan@amazon.com>
1 sibling, 0 replies; 40+ messages in thread
From: Masahiko Sawada @ 2024-02-05 02:06 UTC (permalink / raw)
To: Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, Feb 3, 2024 at 12:12 PM Callahan, Drew <callaan@amazon.com> wrote:
>
> > I think we can prepare an isolation test of the above steps and
> > include it in contrib/test_decoding/specs.
>
> Thanks for the suggestion. Included an isolation test for the
> exact repro and a couple other tests for fully before and fully after.
> These two tests don't add a whole lot of value since this
> is already heavily tested elsewhere, so I'm good with removing.
> However, thought it made sense to include from a perspective of
> totality given the test name.
Thank you for the patch for the isolation test! Since Vignesh also
shared the isolation patch I've merged them and modified some
comments. I've attached the patch. I'm going to merge the isolation
test patch to the patch to fix the issue after we get a consensus on
how to fix it.
>
> > Another simple way to fix this is that we always set
> > need_full_snapshot to true when slot creation. But it would then make
> > the snapshot builder include non-catalog-change transactions too while
> > finding the initial startpoint. Which is not necessary.
>
> Since we're not accumulating changes anyway and we never distribute snapshots
> while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
> This seems pretty contained and wouldn't require the use of a global variable
> and memory context callbacks which is a bit nicer from a readability standpoint.
> It's a little inefficient with memory and CPU, especially in the presence of
> long running transactions, but likely would not be noticeable to most users.
>
> > As for the proposed patch, it uses a global variable, InCreate, and
> > memory context callback to reset it. While I believe this is for not
> > breaking any ABI as this bug exists on back branches too, I think it's
> > better to consider how to fix it in HEAD and then how to backpatch it.
> > We don't necessarily need to have the same fixes for all branches all
> > the time.
>
> Attached a patch that is ABI breaking, but removes the need for a global variable.
> This relies on pushing the context of whether we are in a create mode to the
> logical decoding context, which is cleaner, but not as clean as just pushing
> all the way down to the snapbuilder. I would push down to the snap builder,
> but the entire object is persisted to disk. This is problematic to me since the
> create flag would be persisted causing exisiting slots to restore a snapbuild
> that could have the create flag set to true. This is not problematic as is,
> since the variable would only be leveraged in SnapBuildRestore() which is only
> called while inconsistent, but seems like a future bug waiting to happen.
But IIUC in SnapBuildRestore(), we selectively restore variables from
serialized SnapBuild. For instance, building_full_snapshot and
initial_xmin_horizon are serialized to the disk but ignored when
restoring a snapshot. I think that even if we add the in_create flag
to SnapBuild, it would be ignored on snapshot restore.
I think that it's cleaner if we could pass in_creat flag to
AllocateSnapshotBuilder() than passing the flag down to
SnapBuildFindSnapshot() from standby_decode().
>
> We could always clear the flag when persisting, but that seems a bit hacky.
> We could also fork the snap builder into on-disk and in-memory structs, which
> would be convenient since we would not need to invalidate exisiting snapshots.
> However, not sure how people feel about the fork.
>
> > IIUC we need to skip snapshot restore only when we're finding the
> > initial start point. So probably we don't need to set InCreate when
> > calling create_logical_replication_slot() with find_startpoint =
> > false.
>
> Yep, that's correct. We can safely skip setting InCreate. We could move
> the setter function slotfuncs.c as one way of doing this. I lean towards
> setting it no matter what since it is harmless if we never try to find
> a decoding point and removes the need for future callers of
> CreateInitDecodingContext() from having to determine if it's safe to be false.
> Though I agree that InCreate may a bigger hammer than needed which may need
> to be modified in the future.
I agree to set the flag no matter what. The flag is needed for finding
the initial startpoint but it would be harmless to set it even if the
caller doesn't call DecodingContextFindStartpoint(). If the caller
skips to call DecodingContextFindStartpoint() it's the caller's
responsibility to set a sane LSNs to the slots before staring the
logical decoding. So it makes sense to me to remove the need for
callers to determine to set the flag, as you mentioned.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] isolation_test_v2.patch (5.2K, ../../CAD21AoCsSkwcAYgcUMi2zKp8F=moJL4K1beGD6AV_=pHhfAM9Q@mail.gmail.com/2-isolation_test_v2.patch)
download | inline diff:
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..99dd7005cf
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,44 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-05 02:12 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 0 replies; 40+ messages in thread
From: Masahiko Sawada @ 2024-02-05 02:12 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, Feb 3, 2024 at 10:48 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 2 Feb 2024 at 06:24, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Hi,
> >
> > Thank you for the report!
> >
> > On Fri, Feb 2, 2024 at 7:28 AM Callahan, Drew <callaan@amazon.com> wrote:
> > >
> > > Hello,
> > >
> > > We discovered a race condition during logical replication slot creation that can result in the changes for transactions running at the time of the slot creation to only be partially replicated. We found the cause was due to the slot transitioning from an inconsistent or partially consistent state to a fully consistent state when restoring a snapshot that had been persisted to disk by a different logical slot. We provide a simple reproduction of this issue below:
> > >
> > > Session 1:
> > >
> > > SELECT pg_create_logical_replication_slot('slot1', 'test_decoding');
> > >
> > > CREATE TABLE test (a int);
> > > BEGIN;
> > > INSERT INTO test VALUES (1);
> > >
> > > Session 2:
> > >
> > > SELECT pg_create_logical_replication_slot('slot2', 'test_decoding');
> > >
> > > <query hangs>
> > >
> > > Session 3:
> > >
> > > CHECKPOINT;
> > >
> > > select pg_logical_slot_get_changes('slot1', NULL, NULL);
> > >
> > > <should return nothing of interest>
> > >
> > > Session 1:
> > >
> > > INSERT INTO test VALUES (2);
> > > COMMIT;
> > >
> > > <Session 2 query no longer hangs and successfully creates the slot2>
> > >
> > > Session 2:
> > >
> > > select pg_logical_slot_get_changes('slot1', NULL, NULL);
> > >
> > > select pg_logical_slot_get_changes('slot2', NULL, NULL);
> > >
> > > <expected: no rows of the txn are returned for slot2>
> > > <actual: The 2nd row of the txn is returned for slot2>
> > >
> >
> > I've confirmed this happens HEAD and all supported backbranches. And I
> > think it's a (long-standing) bug.
> >
> > I think we can prepare an isolation test of the above steps and
> > include it in contrib/test_decoding/specs.
> >
> > > Newly created logical replication slots initialize their restart LSN to the current insert position within the WAL and also force a checkpoint to get the current state of the running transactions on the system. This create process will then wait for all of the transactions within that running xact record to complete before being able to transition to the next snapbuild state. During this time period, if another running xact record is written and then a different logical replication process decodes this running xact record, a globally accessible snapshot will be persisted to disk.
> > >
> > > Once all of the transactions from the initial running xact have finished, the process performing the slot creation will become unblocked and will then consume the new running xact record. The process will see a valid snapshot, restore that snapshot from disk, and then transition immediately to the consistent state. The slot will then set the confirmed flush LSN of the slot to the start of the next record after that running xact.
> >
> > My analysis is the same. After the slot creation, the slot's LSNs become like:
> >
> > (tx-begin) < (tx's some changes) < slot's resetart_lsn < (fx's further
> > some changes) < slot's confirmed_flush_lsn < (tx-commit)
> >
> > > We now have a logical replication slot that has a restart LSN after the changes of transactions that will commit after our confirmed flushed LSN in any case where the two running xact records are not fully disjointed. Once those transactions commit, we will then partially stream their changes.
> > >
> > > The attached fix addresses the issue by providing the snapshot builder with enough context to understand whether it is building a snapshot for a slot for the first time or if this is a previously existing slot. This is similar to the “need_full_snapshot” parameter which is already set by the caller to control when and how the snapbuilder is allowed to become consistent.
> > >
> > > With this context, the snapshot builder can always skip performing snapshot restore in order to become fully consistent. Since this issue only occurs when the the logical replication slot consumes a persisted snapshot to become fully consistent we can prevent the issue by disallowing this behavior.
> >
> > I basically agree with the basic idea of skipping snapshot restore
> > while finding the initial start point. But I'd like to hear other
> > opinions too.
> >
> > Another simple way to fix this is that we always set
> > need_full_snapshot to true when slot creation. But it would then make
> > the snapshot builder include non-catalog-change transactions too while
> > finding the initial startpoint. Which is not necessary.
> >
> > As for the proposed patch, it uses a global variable, InCreate, and
> > memory context callback to reset it. While I believe this is for not
> > breaking any ABI as this bug exists on back branches too, I think it's
> > better to consider how to fix it in HEAD and then how to backpatch it.
> > We don't necessarily need to have the same fixes for all branches all
> > the time.
>
> I agree to having a separate patch for HEAD in this case. How about a
> fix like a patch attached.
Thank you for the patch!
I've just sent some comments on how to fix this issue and updated the
isolation test patch[1]. While I basically agree to add the flag in
SnapBuild, I think we can set the flag in AllocateSnapshotBuilder().
Also, we need to bump SNAPBUILD_VERSION.
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoCsSkwcAYgcUMi2zKp8F%3DmoJL4K1beGD6AV_%3DpHhfAM9Q%40mail...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-08 05:54 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Callahan, Drew <callaan@amazon.com>
1 sibling, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-02-08 05:54 UTC (permalink / raw)
To: Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, Feb 3, 2024 at 12:12 PM Callahan, Drew <callaan@amazon.com> wrote:
>
> > I think we can prepare an isolation test of the above steps and
> > include it in contrib/test_decoding/specs.
>
> Thanks for the suggestion. Included an isolation test for the
> exact repro and a couple other tests for fully before and fully after.
> These two tests don't add a whole lot of value since this
> is already heavily tested elsewhere, so I'm good with removing.
> However, thought it made sense to include from a perspective of
> totality given the test name.
>
> > Another simple way to fix this is that we always set
> > need_full_snapshot to true when slot creation. But it would then make
> > the snapshot builder include non-catalog-change transactions too while
> > finding the initial startpoint. Which is not necessary.
>
> Since we're not accumulating changes anyway and we never distribute snapshots
> while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
> This seems pretty contained and wouldn't require the use of a global variable
> and memory context callbacks which is a bit nicer from a readability standpoint.
Sorry I missed this comment.
I think it's a good point that this fix doesn't lead to any on-disk
compatibility while not affecting existing users much, especially for
back-patching. If we want to choose this approach for bank branches,
we need to carefully consider the side impacts.
> It's a little inefficient with memory and CPU, especially in the presence of
> long running transactions, but likely would not be noticeable to most users.
This matches my analysis.
Here is the summary of several proposals we've discussed:
a) Have CreateInitDecodingContext() always pass need_full_snapshot =
true to AllocateSnapshotBuilder().
This is the simplest approach and doesn't break the on-disk
SnapBuildOnDisck compatibility. A downside is that it's a little
inefficient with memory and CPU since it includes non-catalog change
transactions too. Also, passing need_full_snapshot to
CreateInitDecodingContext() will no longer have any meaning.
b) Have snapbuild.c being able to handle multiple SnapBuildOnDisk versions.
This bumps SNAPBUILD_VERSION and therefore breaks the on-disk
SnapBuildOnDisk compatibility, but snapbuild.c can handle multiple
versions. But since different branches are using different versions of
SnapBuildOnDisk, it would probably be hard to have the SnapBuildOnDisk
version that is consistent across all versions.
c) Add a global variable, say in_create, to snapbuild.c
it would also require adding a setter function for in_create. There
are several ideas where to set/reset the flag. One idea is that we
reset the flag in AllocateSnapshotBuilder() and set the flag anywhere
before starting to find the start point, for example at the beginning
of DecodingContextFindStartpoint(). It probably won't require a memory
context callback to make sure to clear the flag. This idea doesn't
have a downside from users and extensions perspects. But I'm slightly
hesitant to add a global variable.
What do you think? and any other ideas?
As a separate topic, I think that this backpatching complexity comes
from the fact that we're serializing a whole SnapBuild to the disk
although we don't need to serialize all of its fields. In order to
make future back-patching easy, it might be worth considering
decoupling the fields that need to be serialized from SnapBuild
struct.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-02-19 09:12 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 40+ messages in thread
From: Masahiko Sawada @ 2024-02-19 09:12 UTC (permalink / raw)
To: Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Feb 8, 2024 at 2:54 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Sat, Feb 3, 2024 at 12:12 PM Callahan, Drew <callaan@amazon.com> wrote:
> >
> > > I think we can prepare an isolation test of the above steps and
> > > include it in contrib/test_decoding/specs.
> >
> > Thanks for the suggestion. Included an isolation test for the
> > exact repro and a couple other tests for fully before and fully after.
> > These two tests don't add a whole lot of value since this
> > is already heavily tested elsewhere, so I'm good with removing.
> > However, thought it made sense to include from a perspective of
> > totality given the test name.
> >
> > > Another simple way to fix this is that we always set
> > > need_full_snapshot to true when slot creation. But it would then make
> > > the snapshot builder include non-catalog-change transactions too while
> > > finding the initial startpoint. Which is not necessary.
> >
> > Since we're not accumulating changes anyway and we never distribute snapshots
> > while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
> > This seems pretty contained and wouldn't require the use of a global variable
> > and memory context callbacks which is a bit nicer from a readability standpoint.
>
> Sorry I missed this comment.
>
> I think it's a good point that this fix doesn't lead to any on-disk
> compatibility while not affecting existing users much, especially for
> back-patching. If we want to choose this approach for bank branches,
> we need to carefully consider the side impacts.
>
> > It's a little inefficient with memory and CPU, especially in the presence of
> > long running transactions, but likely would not be noticeable to most users.
>
> This matches my analysis.
>
> Here is the summary of several proposals we've discussed:
>
> a) Have CreateInitDecodingContext() always pass need_full_snapshot =
> true to AllocateSnapshotBuilder().
>
> This is the simplest approach and doesn't break the on-disk
> SnapBuildOnDisck compatibility. A downside is that it's a little
> inefficient with memory and CPU since it includes non-catalog change
> transactions too. Also, passing need_full_snapshot to
> CreateInitDecodingContext() will no longer have any meaning.
>
> b) Have snapbuild.c being able to handle multiple SnapBuildOnDisk versions.
>
> This bumps SNAPBUILD_VERSION and therefore breaks the on-disk
> SnapBuildOnDisk compatibility, but snapbuild.c can handle multiple
> versions. But since different branches are using different versions of
> SnapBuildOnDisk, it would probably be hard to have the SnapBuildOnDisk
> version that is consistent across all versions.
>
> c) Add a global variable, say in_create, to snapbuild.c
>
> it would also require adding a setter function for in_create. There
> are several ideas where to set/reset the flag. One idea is that we
> reset the flag in AllocateSnapshotBuilder() and set the flag anywhere
> before starting to find the start point, for example at the beginning
> of DecodingContextFindStartpoint(). It probably won't require a memory
> context callback to make sure to clear the flag. This idea doesn't
> have a downside from users and extensions perspects. But I'm slightly
> hesitant to add a global variable.
>
> What do you think? and any other ideas?
I've drafted the idea (c) for discussion (for master and v16 for now).
I also liked the idea (a) but I'm concerned a bit about future impact.
>
> As a separate topic, I think that this backpatching complexity comes
> from the fact that we're serializing a whole SnapBuild to the disk
> although we don't need to serialize all of its fields. In order to
> make future back-patching easy, it might be worth considering
> decoupling the fields that need to be serialized from SnapBuild
> struct.
If we implement this idea, we won't need to bump SNAPBUILD_VERSION for HEAD.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] master_fix_snapbuild_bug_v3.patch (5.3K, ../../CAD21AoBJcRiVBT5_ys13wA9c-8zbbqQJsMiVELX3t+HREwJwbA@mail.gmail.com/2-master_fix_snapbuild_bug_v3.patch)
download | inline diff:
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index ca09c683f1..96e841d77f 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -639,6 +639,12 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
elog(DEBUG1, "searching for logical decoding starting point, starting at %X/%X",
LSN_FORMAT_ARGS(slot->data.restart_lsn));
+ /*
+ * Tell snapshot builder to start finding the start point, preventing
+ * it from restoring snapshots.
+ */
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, true);
+
/* Wait for a consistent starting point */
for (;;)
{
@@ -661,6 +667,8 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
CHECK_FOR_INTERRUPTS();
}
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, false);
+
SpinLockAcquire(&slot->mutex);
slot->data.confirmed_flush = ctx->reader->EndRecPtr;
if (slot->data.two_phase)
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index a0b7947d2f..be2ec367ff 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -190,6 +190,15 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * True if we are finding the logical decoding start point. While finding
+ * the start point, we need to prevent the snapshot being restored
+ * from disk and require all in-progress transactions to finish, in order
+ * to become the consistent state. Otherwise the slot's restart and
+ * confirmed flush LSNs could be initialized to unsafe values.
+ */
+ bool finding_start_point;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -427,6 +436,15 @@ SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr)
builder->two_phase_at = ptr;
}
+/*
+ * Set we are finding the logical decoding start point.
+ */
+void
+SnapBuildFindingStartPoint(SnapBuild *builder, bool finding)
+{
+ builder->finding_start_point = finding;
+}
+
/*
* Should the contents of transaction ending at 'ptr' be decoded?
*/
@@ -1331,7 +1349,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* snapshot to disk that we can use. Can't use this method for the
* initial snapshot when slot is being created and needs full snapshot
* for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * modifying transactions. We cannot use this method also when finding
+ * the logical decoding start point, as it requires all in-progress
+ * transaction to finish.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1396,8 +1416,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /*
+ * b) valid on disk state and neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->finding_start_point &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1581,7 +1605,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index fbdf362396..83d35bc61a 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -90,5 +90,6 @@ extern void SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid,
extern void SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn,
struct xl_running_xacts *running);
extern void SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn);
+extern void SnapBuildFindingStartPoint(SnapBuild *builder, bool finding);
#endif /* SNAPBUILD_H */
[application/octet-stream] v16_fix_snapbuild_bug_v3.patch (5.3K, ../../CAD21AoBJcRiVBT5_ys13wA9c-8zbbqQJsMiVELX3t+HREwJwbA@mail.gmail.com/3-v16_fix_snapbuild_bug_v3.patch)
download | inline diff:
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 7b05cc25a3..2dd3ede41b 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..b57fc33cbb 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -637,6 +637,12 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
elog(DEBUG1, "searching for logical decoding starting point, starting at %X/%X",
LSN_FORMAT_ARGS(slot->data.restart_lsn));
+ /*
+ * Tell snapshot builder to start finding the start point, preventing
+ * it from restoring snapshots.
+ */
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, true);
+
/* Wait for a consistent starting point */
for (;;)
{
@@ -659,6 +665,8 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
CHECK_FOR_INTERRUPTS();
}
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, false);
+
SpinLockAcquire(&slot->mutex);
slot->data.confirmed_flush = ctx->reader->EndRecPtr;
if (slot->data.two_phase)
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..e8dc7acbf3 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -278,6 +278,15 @@ struct SnapBuild
static ResourceOwner SavedResourceOwnerDuringExport = NULL;
static bool ExportInProgress = false;
+/*
+ * True if we are finding the logical decoding start point. While finding
+ * the start point, we need to prevent the snapshot being restored
+ * from disk and require all in-progress transactions to finish, in order
+ * to become the consistent state. Otherwise the slot's restart and
+ * confirmed flush LSNs could be initialized to unsafe values.
+ */
+static bool FindingStartPoint = false;
+
/* ->committed and ->catchange manipulation */
static void SnapBuildPurgeOlderTxn(SnapBuild *builder);
@@ -346,6 +355,8 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
+ FindingStartPoint = false;
+
MemoryContextSwitchTo(oldcontext);
return builder;
@@ -422,6 +433,15 @@ SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr)
builder->two_phase_at = ptr;
}
+/*
+ * Set we are finding the logical decoding start point.
+ */
+void
+SnapBuildFindingStartPoint(SnapBuild *builder, bool finding)
+{
+ FindingStartPoint = finding;
+}
+
/*
* Should the contents of transaction ending at 'ptr' be decoded?
*/
@@ -1326,7 +1346,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* snapshot to disk that we can use. Can't use this method for the
* initial snapshot when slot is being created and needs full snapshot
* for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * modifying transactions. We cannot use this method also when finding
+ * the logical decoding start point, as it requires all in-progress
+ * transaction to finish.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1391,8 +1413,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /*
+ * b) valid on disk state and neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !FindingStartPoint &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index f49b941b53..8239d6dbb3 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -90,5 +90,6 @@ extern void SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid,
extern void SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn,
struct xl_running_xacts *running);
extern void SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn);
+extern void SnapBuildFindingStartPoint(SnapBuild *builder, bool finding);
#endif /* SNAPBUILD_H */
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-13 09:34 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-13 09:34 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear hackers,
While analyzing another failure [1], I found here. I think they occurred by the
same reason.
The reported failure occurred when the replication slot is created in the middle
of the transaction and it reuses the snapshot from other slot. The reproducer is:
```
Session0
SELECT pg_create_logical_replication_slot('slot0', 'test_decoding');
BEGIN;
INSERT INTO foo ...
Session1
SELECT pg_create_logical_replication_slot('slot1', 'test_decoding');
Session2
CHECKPOINT;
SELECT pg_logical_slot_get_changes('slot0', NULL, NULL);
Session0
INSERT INTO var ... // var is defined with (user_catalog_table = true)
COMMIT;
Session1
SELECT pg_logical_slot_get_changes('slot1', NULL, NULL);
-> Assertion failure.
```
> Here is the summary of several proposals we've discussed:
> a) Have CreateInitDecodingContext() always pass need_full_snapshot =
> true to AllocateSnapshotBuilder().
> b) Have snapbuild.c being able to handle multiple SnapBuildOnDisk versions.
> c) Add a global variable, say in_create, to snapbuild.c
Regarding three options raised by Sawada-san, I preferred the approach a).
Since the issue could happen for all supported branches, we should choose the
conservative approach. Also, it is quite painful if there are some codes for
handling the same issue.
Attached patch implemented the approach a) since no one made. I also added
the test which can do assertion failure, but not sure it should be included.
[1]: https://www.postgresql.org/message-id/TYCPR01MB1207717063D701F597EF98A0CF5272%40TYCPR01MB12077.jpnpr...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] master_0001-fix-snapbuild-bug-by-approach-a.patch (13.2K, ../../TYCPR01MB1207719C811F580A8774C79B7F52A2@TYCPR01MB12077.jpnprd01.prod.outlook.com/2-master_0001-fix-snapbuild-bug-by-approach-a.patch)
download | inline diff:
From 560c41d3334f7c50f9a1c69e5c5b0d297f8920e0 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 13 Mar 2024 09:28:00 +0000
Subject: [PATCH] fix snapbuild bug by approach a
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 82 +++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 58 +++++++++++++
src/backend/replication/logical/logical.c | 11 +--
src/backend/replication/slotfuncs.c | 1 -
src/backend/replication/walsender.c | 8 +-
src/bin/pg_basebackup/pg_createsubscriber.c | 5 ++
src/include/replication/logical.h | 1 -
9 files changed, 153 insertions(+), 17 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c1848478a5
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,82 @@
+unused step name: s0_savepoint_release
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3c52e5de9e
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,58 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_savepoint_release" { RELEASE SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 51ffb623c0..e1d60f244d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -306,8 +306,6 @@ StartupDecodingContext(List *output_plugin_options,
*
* plugin -- contains the name of the output plugin
* output_plugin_options -- contains options passed to the output plugin
- * need_full_snapshot -- if true, must obtain a snapshot able to read all
- * tables; if false, one that can read only catalogs is acceptable.
* restart_lsn -- if given as invalid, it's this routine's responsibility to
* mark WAL as reserved by setting a convenient restart_lsn for the slot.
* Otherwise, we set for decoding to start from the given LSN without
@@ -327,7 +325,6 @@ StartupDecodingContext(List *output_plugin_options,
LogicalDecodingContext *
CreateInitDecodingContext(const char *plugin,
List *output_plugin_options,
- bool need_full_snapshot,
XLogRecPtr restart_lsn,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
@@ -420,13 +417,12 @@ CreateInitDecodingContext(const char *plugin,
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(false);
SpinLockAcquire(&slot->mutex);
slot->effective_catalog_xmin = xmin_horizon;
slot->data.catalog_xmin = xmin_horizon;
- if (need_full_snapshot)
- slot->effective_xmin = xmin_horizon;
+ slot->effective_xmin = xmin_horizon;
SpinLockRelease(&slot->mutex);
ReplicationSlotsComputeRequiredXmin(true);
@@ -437,7 +433,8 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ true, /* need_full_snapshot */
+ false, /* fast_forward */
xl_routine, prepare_write, do_write,
update_progress);
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ad79e1fccd..6fb0fd0010 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -148,7 +148,6 @@ create_logical_replication_slot(char *name, char *plugin,
* this point that the output plugin is validated.
*/
ctx = CreateInitDecodingContext(plugin, NIL,
- false, /* just catalogs is OK */
restart_lsn,
XL_ROUTINE(.page_read = read_local_xlog_page,
.segment_open = wal_segment_open,
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..e2924ba91a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1237,7 +1237,6 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
else
{
LogicalDecodingContext *ctx;
- bool need_full_snapshot = false;
Assert(cmd->kind == REPLICATION_KIND_LOGICAL);
@@ -1265,8 +1264,6 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
/*- translator: %s is a CREATE_REPLICATION_SLOT statement */
(errmsg("%s must not be called inside a transaction",
"CREATE_REPLICATION_SLOT ... (SNAPSHOT 'export')")));
-
- need_full_snapshot = true;
}
else if (snapshot_action == CRS_USE_SNAPSHOT)
{
@@ -1298,12 +1295,9 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
/*- translator: %s is a CREATE_REPLICATION_SLOT statement */
(errmsg("%s must not be called in a subtransaction",
"CREATE_REPLICATION_SLOT ... (SNAPSHOT 'use')")));
-
- need_full_snapshot = true;
}
- ctx = CreateInitDecodingContext(cmd->plugin, NIL, need_full_snapshot,
- InvalidXLogRecPtr,
+ ctx = CreateInitDecodingContext(cmd->plugin, NIL, InvalidXLogRecPtr,
XL_ROUTINE(.page_read = logical_read_xlog_page,
.segment_open = WalSndSegmentOpen,
.segment_close = wal_segment_close),
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index c79ed9044c..35a6192817 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -579,6 +579,7 @@ static char *
setup_publisher(struct LogicalRepInfo *dbinfo)
{
char *lsn = NULL;
+ PGconn *conn;
for (int i = 0; i < num_dbs; i++)
{
@@ -654,6 +655,10 @@ setup_publisher(struct LogicalRepInfo *dbinfo)
disconnect_database(conn, false);
}
+ conn = connect_database(dbinfo[0].pubconninfo, true);
+ PQclear((conn, "SELECT * FROM pg_switch_wal()"));
+ disconnect_database(conn, false);
+
return lsn;
}
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index dc2df4ce92..f1b40b4b92 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -119,7 +119,6 @@ extern void CheckLogicalDecodingRequirements(void);
extern LogicalDecodingContext *CreateInitDecodingContext(const char *plugin,
List *output_plugin_options,
- bool need_full_snapshot,
XLogRecPtr restart_lsn,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
--
2.43.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-13 09:47 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-13 09:47 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
> Attached patch implemented the approach a) since no one made. I also added
> the test which can do assertion failure, but not sure it should be included.
Attached one had unnecessary changes. PSA the corrected version.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] master_0001-fix-snapbuild-bug-by-approach-a.patch (12.4K, ../../TYCPR01MB120776B6D38AE1EA49AF07BAEF52A2@TYCPR01MB12077.jpnprd01.prod.outlook.com/2-master_0001-fix-snapbuild-bug-by-approach-a.patch)
download | inline diff:
From d341305e072b64595b28d6550f51c513dd47a634 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 13 Mar 2024 07:30:05 +0000
Subject: [PATCH] fix snapbuild bug by approach a
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 81 +++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 57 +++++++++++++
src/backend/replication/logical/logical.c | 11 +--
src/backend/replication/slotfuncs.c | 1 -
src/backend/replication/walsender.c | 8 +-
src/include/replication/logical.h | 1 -
8 files changed, 146 insertions(+), 17 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..cb0040660b
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,81 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..6cb6c83ad8
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,57 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 51ffb623c0..e1d60f244d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -306,8 +306,6 @@ StartupDecodingContext(List *output_plugin_options,
*
* plugin -- contains the name of the output plugin
* output_plugin_options -- contains options passed to the output plugin
- * need_full_snapshot -- if true, must obtain a snapshot able to read all
- * tables; if false, one that can read only catalogs is acceptable.
* restart_lsn -- if given as invalid, it's this routine's responsibility to
* mark WAL as reserved by setting a convenient restart_lsn for the slot.
* Otherwise, we set for decoding to start from the given LSN without
@@ -327,7 +325,6 @@ StartupDecodingContext(List *output_plugin_options,
LogicalDecodingContext *
CreateInitDecodingContext(const char *plugin,
List *output_plugin_options,
- bool need_full_snapshot,
XLogRecPtr restart_lsn,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
@@ -420,13 +417,12 @@ CreateInitDecodingContext(const char *plugin,
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(false);
SpinLockAcquire(&slot->mutex);
slot->effective_catalog_xmin = xmin_horizon;
slot->data.catalog_xmin = xmin_horizon;
- if (need_full_snapshot)
- slot->effective_xmin = xmin_horizon;
+ slot->effective_xmin = xmin_horizon;
SpinLockRelease(&slot->mutex);
ReplicationSlotsComputeRequiredXmin(true);
@@ -437,7 +433,8 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ true, /* need_full_snapshot */
+ false, /* fast_forward */
xl_routine, prepare_write, do_write,
update_progress);
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ad79e1fccd..6fb0fd0010 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -148,7 +148,6 @@ create_logical_replication_slot(char *name, char *plugin,
* this point that the output plugin is validated.
*/
ctx = CreateInitDecodingContext(plugin, NIL,
- false, /* just catalogs is OK */
restart_lsn,
XL_ROUTINE(.page_read = read_local_xlog_page,
.segment_open = wal_segment_open,
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..e2924ba91a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1237,7 +1237,6 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
else
{
LogicalDecodingContext *ctx;
- bool need_full_snapshot = false;
Assert(cmd->kind == REPLICATION_KIND_LOGICAL);
@@ -1265,8 +1264,6 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
/*- translator: %s is a CREATE_REPLICATION_SLOT statement */
(errmsg("%s must not be called inside a transaction",
"CREATE_REPLICATION_SLOT ... (SNAPSHOT 'export')")));
-
- need_full_snapshot = true;
}
else if (snapshot_action == CRS_USE_SNAPSHOT)
{
@@ -1298,12 +1295,9 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
/*- translator: %s is a CREATE_REPLICATION_SLOT statement */
(errmsg("%s must not be called in a subtransaction",
"CREATE_REPLICATION_SLOT ... (SNAPSHOT 'use')")));
-
- need_full_snapshot = true;
}
- ctx = CreateInitDecodingContext(cmd->plugin, NIL, need_full_snapshot,
- InvalidXLogRecPtr,
+ ctx = CreateInitDecodingContext(cmd->plugin, NIL, InvalidXLogRecPtr,
XL_ROUTINE(.page_read = logical_read_xlog_page,
.segment_open = WalSndSegmentOpen,
.segment_close = wal_segment_close),
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index dc2df4ce92..f1b40b4b92 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -119,7 +119,6 @@ extern void CheckLogicalDecodingRequirements(void);
extern LogicalDecodingContext *CreateInitDecodingContext(const char *plugin,
List *output_plugin_options,
- bool need_full_snapshot,
XLogRecPtr restart_lsn,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
--
2.43.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-18 09:08 Amit Kapila <amit.kapila16@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 2 replies; 40+ messages in thread
From: Amit Kapila @ 2024-03-18 09:08 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Mar 13, 2024 at 3:17 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> > Attached patch implemented the approach a) since no one made. I also added
> > the test which can do assertion failure, but not sure it should be included.
>
I feel setting "needs_full_snapshot" to true for decoding means the
snapshot will start tracking non-catalog committed xacts as well which
is costly. See SnapBuildCommitTxn(). Can we avoid this problem if we
would have list of all running xacts when we serialize the snapshot by
not decoding any xact whose xid lies in that list? If so, one idea to
achieve could be that we maintain the highest_running_xid while
serailizing the snapshot and then during restore if that
highest_running_xid is <= builder->initial_xmin_horizon, then we
ignore restoring the snapshot. We already have few such cases handled
in SnapBuildRestore().
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-18 09:18 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-03-18 09:18 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Feb 19, 2024 at 2:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Feb 8, 2024 at 2:54 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Sat, Feb 3, 2024 at 12:12 PM Callahan, Drew <callaan@amazon.com> wrote:
> > >
> > > > I think we can prepare an isolation test of the above steps and
> > > > include it in contrib/test_decoding/specs.
> > >
> > > Thanks for the suggestion. Included an isolation test for the
> > > exact repro and a couple other tests for fully before and fully after.
> > > These two tests don't add a whole lot of value since this
> > > is already heavily tested elsewhere, so I'm good with removing.
> > > However, thought it made sense to include from a perspective of
> > > totality given the test name.
> > >
> > > > Another simple way to fix this is that we always set
> > > > need_full_snapshot to true when slot creation. But it would then make
> > > > the snapshot builder include non-catalog-change transactions too while
> > > > finding the initial startpoint. Which is not necessary.
> > >
> > > Since we're not accumulating changes anyway and we never distribute snapshots
> > > while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
> > > This seems pretty contained and wouldn't require the use of a global variable
> > > and memory context callbacks which is a bit nicer from a readability standpoint.
> >
> > Sorry I missed this comment.
> >
> > I think it's a good point that this fix doesn't lead to any on-disk
> > compatibility while not affecting existing users much, especially for
> > back-patching. If we want to choose this approach for bank branches,
> > we need to carefully consider the side impacts.
> >
> > > It's a little inefficient with memory and CPU, especially in the presence of
> > > long running transactions, but likely would not be noticeable to most users.
> >
> > This matches my analysis.
> >
> > Here is the summary of several proposals we've discussed:
> >
> > a) Have CreateInitDecodingContext() always pass need_full_snapshot =
> > true to AllocateSnapshotBuilder().
> >
> > This is the simplest approach and doesn't break the on-disk
> > SnapBuildOnDisck compatibility. A downside is that it's a little
> > inefficient with memory and CPU since it includes non-catalog change
> > transactions too. Also, passing need_full_snapshot to
> > CreateInitDecodingContext() will no longer have any meaning.
> >
> > b) Have snapbuild.c being able to handle multiple SnapBuildOnDisk versions.
> >
> > This bumps SNAPBUILD_VERSION and therefore breaks the on-disk
> > SnapBuildOnDisk compatibility, but snapbuild.c can handle multiple
> > versions. But since different branches are using different versions of
> > SnapBuildOnDisk, it would probably be hard to have the SnapBuildOnDisk
> > version that is consistent across all versions.
> >
> > c) Add a global variable, say in_create, to snapbuild.c
> >
> > it would also require adding a setter function for in_create. There
> > are several ideas where to set/reset the flag. One idea is that we
> > reset the flag in AllocateSnapshotBuilder() and set the flag anywhere
> > before starting to find the start point, for example at the beginning
> > of DecodingContextFindStartpoint(). It probably won't require a memory
> > context callback to make sure to clear the flag. This idea doesn't
> > have a downside from users and extensions perspects. But I'm slightly
> > hesitant to add a global variable.
> >
> > What do you think? and any other ideas?
>
> I've drafted the idea (c) for discussion (for master and v16 for now).
> I also liked the idea (a) but I'm concerned a bit about future impact.
>
I agree that (c) would be better than (a) as it avoids collecting
non-catalog xacts in snapshot. However, I think we shouldn't avoid
restoring the snapshot unless really required. See my previous
response.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-19 02:16 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-19 02:16 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Amit,
> I feel setting "needs_full_snapshot" to true for decoding means the
> snapshot will start tracking non-catalog committed xacts as well which
> is costly.
I think the approach was most conservative one which does not have to change
the version of the snapshot. However, I understood that you wanted to consider
the optimized solution for HEAD first.
> See SnapBuildCommitTxn(). Can we avoid this problem if we
> would have list of all running xacts when we serialize the snapshot by
> not decoding any xact whose xid lies in that list? If so, one idea to
> achieve could be that we maintain the highest_running_xid while
> serailizing the snapshot and then during restore if that
> highest_running_xid is <= builder->initial_xmin_horizon, then we
> ignore restoring the snapshot. We already have few such cases handled
> in SnapBuildRestore().
Based on the idea, I made a prototype. It can pass tests added by others and me.
How do other think?
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] 0001-Serialize-running-xacts.patch (13.3K, ../../TYCPR01MB120774D17B90FFDAC087B959FF52C2@TYCPR01MB12077.jpnprd01.prod.outlook.com/2-0001-Serialize-running-xacts.patch)
download | inline diff:
From 633fd196d13b85b0566669c4bd6ed011b2f26a31 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Tue, 19 Mar 2024 02:08:57 +0000
Subject: [PATCH] Serialize running xacts
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 81 +++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 57 +++++++++++++
src/backend/replication/logical/snapbuild.c | 73 +++++++++++++++--
5 files changed, 208 insertions(+), 7 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..cb0040660b
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,81 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..6cb6c83ad8
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,57 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index ac24b51860..c4f8dae8c9 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -273,6 +273,30 @@ struct SnapBuild
/* This array must be sorted in xidComparator order */
TransactionId *xip;
} catchange;
+
+ /*
+ * Array of concurrently running transactions when the snapshot was
+ * serialized.
+ *
+ * The serialized snapshot can be used even by other replication slots.
+ * However, concurrent transactions might be partially decoded and output
+ * if a newly created slot finds the snapshot. The issue could happen when
+ * slot creation and serialization occur while running the same concurrent
+ * transaction.
+ *
+ * Now, to avoid the problem, we serialize concurrently running
+ * transactions. The array is used to check whether restoring the snapshot
+ * can be done. If one of the listed transactions is older than
+ * initial_xmin_horizon, we skip using the serialized snapshot.
+ */
+ struct
+ {
+ /* number of transactions */
+ size_t xcnt;
+
+ /* Array of transacitons. No need to be ordered. */
+ TransactionId *xip;
+ } running;
};
/*
@@ -302,7 +326,8 @@ static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running
static void SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff);
/* serialization functions */
-static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn);
+static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, int xcnt,
+ TransactionId *xip);
static bool SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn);
static void SnapBuildRestoreContents(int fd, char *dest, Size size, const char *path);
@@ -1231,7 +1256,7 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
return;
}
else
- SnapBuildSerialize(builder, lsn);
+ SnapBuildSerialize(builder, lsn, running->xcnt, running->xids);
/*
* Update range of interesting xids based on the running xacts
@@ -1580,7 +1605,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
@@ -1594,7 +1619,7 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
if (builder->state < SNAPBUILD_CONSISTENT)
SnapBuildRestore(builder, lsn);
else
- SnapBuildSerialize(builder, lsn);
+ SnapBuildSerialize(builder, lsn, 0, NULL);
}
/*
@@ -1602,7 +1627,8 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
* been done by another decoding process.
*/
static void
-SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
+SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, int xcnt,
+ TransactionId *xip)
{
Size needed_length;
SnapBuildOnDisk *ondisk = NULL;
@@ -1701,7 +1727,8 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
catchange_xcnt = dclist_count(&builder->reorder->catchange_txns);
needed_length = sizeof(SnapBuildOnDisk) +
- sizeof(TransactionId) * (builder->committed.xcnt + catchange_xcnt);
+ sizeof(TransactionId) * (builder->committed.xcnt + catchange_xcnt +
+ xcnt);
ondisk_c = palloc0(needed_length);
ondisk = (SnapBuildOnDisk *) ondisk_c;
@@ -1723,6 +1750,8 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
ondisk->builder.catchange.xip = NULL;
/* update catchange only on disk data */
ondisk->builder.catchange.xcnt = catchange_xcnt;
+ /* update running only on disk data */
+ ondisk->builder.running.xcnt = xcnt;
COMP_CRC32C(ondisk->checksum,
&ondisk->builder,
@@ -1746,6 +1775,15 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
ondisk_c += sz;
}
+ /* copy concurrently running xacts */
+ if (xcnt > 0)
+ {
+ sz = sizeof(TransactionId) * xcnt;
+ memcpy(ondisk_c, xip, sz);
+ COMP_CRC32C(ondisk->checksum, ondisk_c, sz);
+ ondisk_c += sz;
+ }
+
FIN_CRC32C(ondisk->checksum);
/* we have valid data now, open tempfile and write it there */
@@ -1920,6 +1958,15 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
COMP_CRC32C(checksum, ondisk.builder.catchange.xip, sz);
}
+ /* restore concurrently running xacts information */
+ if (ondisk.builder.running.xcnt > 0)
+ {
+ sz = sizeof(TransactionId) * ondisk.builder.running.xcnt;
+ ondisk.builder.running.xip = MemoryContextAllocZero(builder->context, sz);
+ SnapBuildRestoreContents(fd, (char *) ondisk.builder.running.xip, sz, path);
+ COMP_CRC32C(checksum, ondisk.builder.running.xip, sz);
+ }
+
if (CloseTransientFile(fd) != 0)
ereport(ERROR,
(errcode_for_file_access(),
@@ -1954,6 +2001,18 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
if (TransactionIdPrecedes(ondisk.builder.xmin, builder->initial_xmin_horizon))
goto snapshot_not_interesting;
+ /*
+ * Don't use a snapshot that one of listed transactions in running has
+ * smaller xid than initial_xmin_horizon.
+ */
+ for (int i = 0; i < ondisk.builder.running.xcnt; i++)
+ {
+ TransactionId xid = ondisk.builder.running.xip[i];
+
+ if (!TransactionIdPrecedes(builder->initial_xmin_horizon, xid))
+ goto snapshot_not_interesting;
+ }
+
/*
* Consistent snapshots have no next phase. Reset next_phase_at as it is
* possible that an old value may remain.
@@ -2007,6 +2066,8 @@ snapshot_not_interesting:
pfree(ondisk.builder.committed.xip);
if (ondisk.builder.catchange.xip != NULL)
pfree(ondisk.builder.catchange.xip);
+ if (ondisk.builder.running.xip != NULL)
+ pfree(ondisk.builder.running.xip);
return false;
}
--
2.43.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-19 02:33 Amit Kapila <amit.kapila16@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-03-19 02:33 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Mar 19, 2024 at 7:46 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> I think the approach was most conservative one which does not have to change
> the version of the snapshot. However, I understood that you wanted to consider
> the optimized solution for HEAD first.
>
Right, let's see if we can have a solution other than always avoiding
restoring snapshots during slot creation even if that is for just
HEAD.
> > See SnapBuildCommitTxn(). Can we avoid this problem if we
> > would have list of all running xacts when we serialize the snapshot by
> > not decoding any xact whose xid lies in that list? If so, one idea to
> > achieve could be that we maintain the highest_running_xid while
> > serailizing the snapshot and then during restore if that
> > highest_running_xid is <= builder->initial_xmin_horizon, then we
> > ignore restoring the snapshot. We already have few such cases handled
> > in SnapBuildRestore().
>
> Based on the idea, I made a prototype. It can pass tests added by others and me.
> How do other think?
>
Won't it be possible to achieve the same thing if we just save
(serialize) the highest xid among all running xacts?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-19 04:39 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-19 04:39 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Amit,
>
> Won't it be possible to achieve the same thing if we just save
> (serialize) the highest xid among all running xacts?
>
Indeed, here is an updated version.
Since the array in xl_running_xact is not ordered, entries of it must be seeked
and found the highest one.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] v2-0001-Serialize-running-xacts.patch (11.6K, ../../TYCPR01MB120771B4C477C9677031F0D9BF52C2@TYCPR01MB12077.jpnprd01.prod.outlook.com/2-v2-0001-Serialize-running-xacts.patch)
download | inline diff:
From 187a61e3a05bb41a8e06acda00c9b5cc6e438907 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Tue, 19 Mar 2024 02:08:57 +0000
Subject: [PATCH v2] Serialize running xacts
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 81 +++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 57 +++++++++++++
src/backend/replication/logical/snapbuild.c | 47 +++++++++--
5 files changed, 183 insertions(+), 6 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..cb0040660b
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,81 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..6cb6c83ad8
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,57 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index ac24b51860..5fc33107d9 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -211,6 +211,23 @@ struct SnapBuild
*/
TransactionId next_phase_at;
+ /*
+ * The highest TransactionId which was running when the snapshot was
+ * serialized.
+ *
+ * The serialized snapshot can be used even by other replication slots.
+ * However, concurrent transactions might be partially decoded and output
+ * if a newly created slot finds the snapshot. The issue could happen when
+ * slot creation and serialization occur while running the same concurrent
+ * transaction.
+ *
+ * Now, to avoid the problem, we serialize a concurrently running
+ * transaction which have the highest TransactionId. The value is used to
+ * check whether restoring the snapshot can be done. If it is older than
+ * initial_xmin_horizon, we skip using the serialized snapshot.
+ */
+ TransactionId highest;
+
/*
* Array of transactions which could have catalog changes that committed
* between xmin and xmax.
@@ -302,7 +319,7 @@ static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running
static void SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff);
/* serialization functions */
-static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn);
+static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, TransactionId highest);
static bool SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn);
static void SnapBuildRestoreContents(int fd, char *dest, Size size, const char *path);
@@ -1231,7 +1248,18 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
return;
}
else
- SnapBuildSerialize(builder, lsn);
+ {
+ TransactionId xid = InvalidTransactionId;
+
+ /* If the concurrent transactions are recorded pass the latest one */
+ for (int i = 0; i < running->xcnt; i++)
+ {
+ if (TransactionIdPrecedes(xid, running->xids[i]))
+ xid = running->xids[i];
+ }
+
+ SnapBuildSerialize(builder, lsn, xid);
+ }
/*
* Update range of interesting xids based on the running xacts
@@ -1580,7 +1608,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
@@ -1594,7 +1622,7 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
if (builder->state < SNAPBUILD_CONSISTENT)
SnapBuildRestore(builder, lsn);
else
- SnapBuildSerialize(builder, lsn);
+ SnapBuildSerialize(builder, lsn, InvalidTransactionId);
}
/*
@@ -1602,7 +1630,7 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
* been done by another decoding process.
*/
static void
-SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
+SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, TransactionId highest)
{
Size needed_length;
SnapBuildOnDisk *ondisk = NULL;
@@ -1715,6 +1743,7 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
ondisk_c += sizeof(SnapBuildOnDisk);
memcpy(&ondisk->builder, builder, sizeof(SnapBuild));
+ ondisk->builder.highest = highest;
/* NULL-ify memory-only data */
ondisk->builder.context = NULL;
ondisk->builder.snapshot = NULL;
@@ -1954,6 +1983,14 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
if (TransactionIdPrecedes(ondisk.builder.xmin, builder->initial_xmin_horizon))
goto snapshot_not_interesting;
+ /*
+ * Don't use a snapshot that one of listed transactions in running has
+ * smaller xid than initial_xmin_horizon.
+ */
+ if (!TransactionIdPrecedes(builder->initial_xmin_horizon,
+ ondisk.builder.highest))
+ goto snapshot_not_interesting;
+
/*
* Consistent snapshots have no next phase. Reset next_phase_at as it is
* possible that an old value may remain.
--
2.43.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-19 16:42 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-03-19 16:42 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Mar 18, 2024 at 6:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> If so, one idea to
> achieve could be that we maintain the highest_running_xid while
> serailizing the snapshot and then during restore if that
> highest_running_xid is <= builder->initial_xmin_horizon, then we
> ignore restoring the snapshot. We already have few such cases handled
> in SnapBuildRestore().
I think that builder->initial_xmin_horizon could be older than
highest_running_xid, for example, when there is a logical replication
slot whose catalog_xmin is old. However, even in this case, we might
need to ignore restoring the snapshot. For example, a slightly
modified test case still can cause the same problem.
The test case in the Kuroda-san's v2 patch:
permutation "s0_init" "s0_begin" "s0_insert1" "s1_init"
"s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
"s1_get_changes_slot0"\ "s1_get_changes_slot1"
Modified-version test case (add "s0_insert1" between "s0_init" and "s0_begin"):
permutation "s0_init" "s0_insert1" "s0_begin" "s0_insert1" "s1_init"
"s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
"s1_get_changes_slot0\ " "s1_get_changes_slot1"
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-27 07:54 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-27 07:54 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
Thanks for giving comments and sorry for late reply.
> > If so, one idea to
> > achieve could be that we maintain the highest_running_xid while
> > serailizing the snapshot and then during restore if that
> > highest_running_xid is <= builder->initial_xmin_horizon, then we
> > ignore restoring the snapshot. We already have few such cases handled
> > in SnapBuildRestore().
>
> I think that builder->initial_xmin_horizon could be older than
> highest_running_xid, for example, when there is a logical replication
> slot whose catalog_xmin is old. However, even in this case, we might
> need to ignore restoring the snapshot. For example, a slightly
> modified test case still can cause the same problem.
>
> The test case in the Kuroda-san's v2 patch:
> permutation "s0_init" "s0_begin" "s0_insert1" "s1_init"
> "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
> "s1_get_changes_slot0"\ "s1_get_changes_slot1"
>
> Modified-version test case (add "s0_insert1" between "s0_init" and "s0_begin"):
> permutation "s0_init" "s0_insert1" "s0_begin" "s0_insert1" "s1_init"
> "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
> "s1_get_changes_slot0\ " "s1_get_changes_slot1"
Good catch, I confirmed that the variant still partially output transactions:
```
step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
data
-----------------------------------------
BEGIN
table public.tbl: INSERT: val1[integer]:2
COMMIT
(3 rows)
```
/////
I analyzed a bit. In below descriptions, I uses these notations:
txn0 - has a single insert
txn1 - has begin-insert-commit operations.
slot1 is created in parallel
While creating slot0, initial_xmin_horizon was also set, and it come from
GetOldestSafeDecodingTransactionId().
Since there were no slots in the database before, the value was just a next xid.
Then, the pinned xid was assigned to the txn0.
It is correct behavior because slot0 must decode txn0.
Finally, while creating slot1 after the txn1, the snapshot serialized by slot0 was
restored once. txn1 was the recorded as the highest_running, but initial_xmin was
older than it. So the snapshot was restored and txn1 was partially decoded.
initial_xmin_horizon (txn0) < highest_running_xid (txn1)
/////
So, how should we fix? One idea is based on (c), and adds a flag which indicates
an existence of concurrent transactions. The restoration is skipped if both two
flags are true. PSA the PoC patch.
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
Attachments:
[application/octet-stream] 0001-Serialize-a-flag-and-use-finding_start_point.patch (15.8K, ../../TYCPR01MB12077ADB13450B422B6161A01F5342@TYCPR01MB12077.jpnprd01.prod.outlook.com/2-0001-Serialize-a-flag-and-use-finding_start_point.patch)
download | inline diff:
From 4c7f9b6d3845e1fe4c80ab0d884ce00eddd6036e Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 27 Mar 2024 07:09:48 +0000
Subject: [PATCH] Serialize a flag and use finding_start_point
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 129 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 60 ++++++++
src/backend/replication/logical/logical.c | 8 ++
src/backend/replication/logical/snapbuild.c | 47 ++++++-
src/include/replication/snapbuild.h | 1 +
7 files changed, 242 insertions(+), 7 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..243d1ee784
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,129 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_insert1 s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+COMMIT
+(3 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..73e6cd7b01
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,60 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
+
+permutation "s0_init" "s0_insert1" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 51ffb623c0..dcf9f841c3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -649,6 +649,12 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
elog(DEBUG1, "searching for logical decoding starting point, starting at %X/%X",
LSN_FORMAT_ARGS(slot->data.restart_lsn));
+ /*
+ * Tell snapshot builder to start finding the start point, preventing
+ * it from restoring snapshots.
+ */
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, true);
+
/* Wait for a consistent starting point */
for (;;)
{
@@ -671,6 +677,8 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
CHECK_FOR_INTERRUPTS();
}
+ SnapBuildFindingStartPoint(ctx->snapshot_builder, false);
+
SpinLockAcquire(&slot->mutex);
slot->data.confirmed_flush = ctx->reader->EndRecPtr;
if (slot->data.two_phase)
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index ac24b51860..95e592b1e0 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,21 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * True if we are finding the logical decoding start point. While finding
+ * the start point, we need to prevent the snapshot being restored
+ * from disk and require all in-progress transactions to finish, in order
+ * to become the consistent state. Otherwise the slot's restart and
+ * confirmed flush LSNs could be initialized to unsafe values.
+ */
+ bool finding_start_point;
+
+ /*
+ * True if there are concurrent transctions while being serialized. This
+ * flag is used with finding_start_point. See comments atop the attribute.
+ */
+ bool is_there_running_xact;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -302,7 +317,7 @@ static bool SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running
static void SnapBuildWaitSnapshot(xl_running_xacts *running, TransactionId cutoff);
/* serialization functions */
-static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn);
+static void SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, bool is_there_running_xact);
static bool SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn);
static void SnapBuildRestoreContents(int fd, char *dest, Size size, const char *path);
@@ -426,6 +441,15 @@ SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr)
builder->two_phase_at = ptr;
}
+/*
+ * Set we are finding the logical decoding start point.
+ */
+void
+SnapBuildFindingStartPoint(SnapBuild *builder, bool finding)
+{
+ builder->finding_start_point = finding;
+}
+
/*
* Should the contents of transaction ending at 'ptr' be decoded?
*/
@@ -1231,7 +1255,7 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
return;
}
else
- SnapBuildSerialize(builder, lsn);
+ SnapBuildSerialize(builder, lsn, running->xcnt > 0);
/*
* Update range of interesting xids based on the running xacts
@@ -1330,7 +1354,9 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* snapshot to disk that we can use. Can't use this method for the
* initial snapshot when slot is being created and needs full snapshot
* for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * modifying transactions. We cannot use this method also when finding
+ * the logical decoding start point, as it requires all in-progress
+ * transaction to finish.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1580,7 +1606,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
@@ -1594,7 +1620,7 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
if (builder->state < SNAPBUILD_CONSISTENT)
SnapBuildRestore(builder, lsn);
else
- SnapBuildSerialize(builder, lsn);
+ SnapBuildSerialize(builder, lsn, false);
}
/*
@@ -1602,7 +1628,7 @@ SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
* been done by another decoding process.
*/
static void
-SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
+SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn, bool is_there_running_xact)
{
Size needed_length;
SnapBuildOnDisk *ondisk = NULL;
@@ -1715,6 +1741,7 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
ondisk_c += sizeof(SnapBuildOnDisk);
memcpy(&ondisk->builder, builder, sizeof(SnapBuild));
+ ondisk->builder.is_there_running_xact = is_there_running_xact;
/* NULL-ify memory-only data */
ondisk->builder.context = NULL;
ondisk->builder.snapshot = NULL;
@@ -1954,6 +1981,14 @@ SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
if (TransactionIdPrecedes(ondisk.builder.xmin, builder->initial_xmin_horizon))
goto snapshot_not_interesting;
+ /*
+ * Don't use a snapshot if we are finding the logical decoding start point
+ * and concurrent transactions occur while the snapshot has been
+ * serialized.
+ */
+ if (builder->finding_start_point && ondisk.builder.is_there_running_xact)
+ goto snapshot_not_interesting;
+
/*
* Consistent snapshots have no next phase. Reset next_phase_at as it is
* possible that an old value may remain.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index fbdf362396..83d35bc61a 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -90,5 +90,6 @@ extern void SnapBuildProcessNewCid(SnapBuild *builder, TransactionId xid,
extern void SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn,
struct xl_running_xacts *running);
extern void SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn);
+extern void SnapBuildFindingStartPoint(SnapBuild *builder, bool finding);
#endif /* SNAPBUILD_H */
--
2.43.0
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-27 08:26 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-03-27 08:26 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Mar 27, 2024 at 4:54 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Sawada-san,
>
> Thanks for giving comments and sorry for late reply.
>
> > > If so, one idea to
> > > achieve could be that we maintain the highest_running_xid while
> > > serailizing the snapshot and then during restore if that
> > > highest_running_xid is <= builder->initial_xmin_horizon, then we
> > > ignore restoring the snapshot. We already have few such cases handled
> > > in SnapBuildRestore().
> >
> > I think that builder->initial_xmin_horizon could be older than
> > highest_running_xid, for example, when there is a logical replication
> > slot whose catalog_xmin is old. However, even in this case, we might
> > need to ignore restoring the snapshot. For example, a slightly
> > modified test case still can cause the same problem.
> >
> > The test case in the Kuroda-san's v2 patch:
> > permutation "s0_init" "s0_begin" "s0_insert1" "s1_init"
> > "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
> > "s1_get_changes_slot0"\ "s1_get_changes_slot1"
> >
> > Modified-version test case (add "s0_insert1" between "s0_init" and "s0_begin"):
> > permutation "s0_init" "s0_insert1" "s0_begin" "s0_insert1" "s1_init"
> > "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit"
> > "s1_get_changes_slot0\ " "s1_get_changes_slot1"
>
> Good catch, I confirmed that the variant still partially output transactions:
>
> ```
> step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
> data
> -----------------------------------------
> BEGIN
> table public.tbl: INSERT: val1[integer]:2
> COMMIT
> (3 rows)
> ```
>
> /////
>
> I analyzed a bit. In below descriptions, I uses these notations:
>
> txn0 - has a single insert
> txn1 - has begin-insert-commit operations.
> slot1 is created in parallel
>
> While creating slot0, initial_xmin_horizon was also set, and it come from
> GetOldestSafeDecodingTransactionId().
> Since there were no slots in the database before, the value was just a next xid.
>
> Then, the pinned xid was assigned to the txn0.
> It is correct behavior because slot0 must decode txn0.
>
> Finally, while creating slot1 after the txn1, the snapshot serialized by slot0 was
> restored once. txn1 was the recorded as the highest_running, but initial_xmin was
> older than it. So the snapshot was restored and txn1 was partially decoded.
>
> initial_xmin_horizon (txn0) < highest_running_xid (txn1)
>
> /////
>
> So, how should we fix? One idea is based on (c), and adds a flag which indicates
> an existence of concurrent transactions. The restoration is skipped if both two
> flags are true. PSA the PoC patch.
>
With the PoC patch, we check ondisk.builder.is_there_running_xact in
SnapBuildRestore(), but can we just check running->xcnt in
SnapBuildFindSnapshot() to skip calling SnapBuildRestore()? That is,
if builder->initial_xmin_horizon is valid (or
builder->finding_start_point is true) and running->xcnt > 0, we skip
the snapshot restore. However, I think there are still cases where we
unnecessarily skip snapshot restores.
Probably, what we would like to avoid is, we compute
initial_xmin_horizon and start to find the initial start point while
there is a concurrently running transaction, and then jump to the
consistent state by restoring the consistent snapshot before the
concurrent transaction commits. So we can ignore snapshot restores if
(oldest XID among transactions running at the time of
CreateInitDecodingContext()) >= (OldestRunningXID in
xl_running_xacts).
I've drafted this idea in the attached patch just for discussion.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] 0001-draft-fix.patch (17.9K, ../../CAD21AoAHWJf10+jRnALY_=nyByE+BgDVV0gnXuiRhxAne+JJdA@mail.gmail.com/2-0001-draft-fix.patch)
download | inline diff:
From 24bdb1df051e33c42f1cb2da86f329eae9d4ff6d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Mon, 25 Mar 2024 11:29:21 +0900
Subject: [PATCH] draft fix.
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 129 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 58 ++++++++
src/backend/replication/logical/logical.c | 13 +-
src/backend/replication/logical/slotsync.c | 3 +-
src/backend/replication/logical/snapbuild.c | 16 ++-
src/backend/storage/ipc/procarray.c | 9 +-
src/include/replication/snapbuild.h | 4 +-
src/include/storage/procarray.h | 3 +-
9 files changed, 228 insertions(+), 10 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..fa4498fb11
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,129 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_insert1 s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+COMMIT
+(3 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..81c6a66039
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,58 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and reach the consistent state until
+# decoding s0-transaction's commit recrod. We check if the get_changes on 'slot1'
+# will not return any s0-transaction's changes as its confirmed_flush_lsn will
+# be after the s0-transaction's commit record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+permutation "s0_init" "s0_insert1" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 51ffb623c0..bf1e9bf4e4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -149,6 +149,7 @@ static LogicalDecodingContext *
StartupDecodingContext(List *output_plugin_options,
XLogRecPtr start_lsn,
TransactionId xmin_horizon,
+ TransactionId oldest_running,
bool need_full_snapshot,
bool fast_forward,
XLogReaderRoutine *xl_routine,
@@ -210,8 +211,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
- AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, oldest_running,
+ start_lsn, need_full_snapshot, slot->data.two_phase_at);
ctx->reorder->private_data = ctx;
@@ -335,6 +336,7 @@ CreateInitDecodingContext(const char *plugin,
LogicalOutputPluginWriterUpdateProgress update_progress)
{
TransactionId xmin_horizon = InvalidTransactionId;
+ TransactionId oldest_running = InvalidTransactionId;
ReplicationSlot *slot;
NameData plugin_name;
LogicalDecodingContext *ctx;
@@ -420,7 +422,8 @@ CreateInitDecodingContext(const char *plugin,
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot,
+ &oldest_running);
SpinLockAcquire(&slot->mutex);
slot->effective_catalog_xmin = xmin_horizon;
@@ -437,6 +440,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
+ oldest_running,
need_full_snapshot, false,
xl_routine, prepare_write, do_write,
update_progress);
@@ -584,7 +588,8 @@ CreateDecodingContext(XLogRecPtr start_lsn,
}
ctx = StartupDecodingContext(output_plugin_options,
- start_lsn, InvalidTransactionId, false,
+ start_lsn, InvalidTransactionId,
+ InvalidTransactionId, false,
fast_forward, xl_routine, prepare_write,
do_write, update_progress);
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 30480960c5..d46b07c8b5 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -592,6 +592,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
{
NameData plugin_name;
TransactionId xmin_horizon = InvalidTransactionId;
+ TransactionId tmp = InvalidTransactionId;
/* Skip creating the local slot if remote_slot is invalidated already */
if (remote_slot->invalidated != RS_INVAL_NONE)
@@ -623,7 +624,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
reserve_wal_for_local_slot(remote_slot->restart_lsn);
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- xmin_horizon = GetOldestSafeDecodingTransactionId(true);
+ xmin_horizon = GetOldestSafeDecodingTransactionId(true, &tmp);
SpinLockAcquire(&slot->mutex);
slot->effective_catalog_xmin = xmin_horizon;
slot->data.catalog_xmin = xmin_horizon;
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index ac24b51860..94def02f11 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -185,6 +185,7 @@ struct SnapBuild
* indicates there are no running xids with an xid smaller than this.
*/
TransactionId initial_xmin_horizon;
+ TransactionId initial_oldest_running;
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
@@ -315,6 +316,7 @@ static void SnapBuildRestoreContents(int fd, char *dest, Size size, const char *
SnapBuild *
AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
+ TransactionId oldest_running,
XLogRecPtr start_lsn,
bool need_full_snapshot,
XLogRecPtr two_phase_at)
@@ -346,6 +348,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->catchange.xip = NULL;
builder->initial_xmin_horizon = xmin_horizon;
+ builder->initial_oldest_running = oldest_running;
builder->start_decoding_at = start_lsn;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
@@ -572,6 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
TransactionId xid;
TransactionId safeXid;
TransactionId *newxip;
+ TransactionId tmp;
int newxcnt = 0;
Assert(XactIsoLevel == XACT_REPEATABLE_READ);
@@ -605,7 +609,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
* the horizon is enforced.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
- safeXid = GetOldestSafeDecodingTransactionId(false);
+ safeXid = GetOldestSafeDecodingTransactionId(false, &tmp);
LWLockRelease(ProcArrayLock);
if (TransactionIdFollows(safeXid, snap->xmin))
@@ -1305,6 +1309,15 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
builder->last_serialized_snapshot);
}
+static bool
+xxx_restore_ok(SnapBuild *builder, xl_running_xacts *running)
+{
+ if (!TransactionIdIsValid(builder->initial_oldest_running))
+ return true;
+
+ return TransactionIdPrecedes(builder->initial_oldest_running,
+ running->oldestRunningXid);
+}
/*
* Build the start of a snapshot that's capable of decoding the catalog.
@@ -1397,6 +1410,7 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
}
/* b) valid on disk state and not building full snapshot */
else if (!builder->building_full_snapshot &&
+ xxx_restore_ok(builder, running) &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb6..5fbdec69ba 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2929,10 +2929,11 @@ GetOldestActiveTransactionId(void)
* that the caller will immediately use the xid to peg the xmin horizon.
*/
TransactionId
-GetOldestSafeDecodingTransactionId(bool catalogOnly)
+GetOldestSafeDecodingTransactionId(bool catalogOnly, TransactionId *oldestActiveXid_p)
{
ProcArrayStruct *arrayP = procArray;
TransactionId oldestSafeXid;
+ TransactionId oldestActiveXid;
int index;
bool recovery_in_progress = RecoveryInProgress();
@@ -2948,6 +2949,7 @@ GetOldestSafeDecodingTransactionId(bool catalogOnly)
*/
LWLockAcquire(XidGenLock, LW_SHARED);
oldestSafeXid = XidFromFullTransactionId(TransamVariables->nextXid);
+ oldestActiveXid = oldestSafeXid;
/*
* If there's already a slot pegging the xmin horizon, we can start with
@@ -2998,11 +3000,16 @@ GetOldestSafeDecodingTransactionId(bool catalogOnly)
if (TransactionIdPrecedes(xid, oldestSafeXid))
oldestSafeXid = xid;
+
+ if (TransactionIdPrecedes(xid, oldestActiveXid))
+ oldestActiveXid = xid;
}
}
LWLockRelease(XidGenLock);
+ *oldestActiveXid_p = oldestActiveXid;
+
return oldestSafeXid;
}
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index fbdf362396..63a380b692 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -60,7 +60,9 @@ struct xl_running_xacts;
extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
- TransactionId xmin_horizon, XLogRecPtr start_lsn,
+ TransactionId xmin_horizon,
+ TransactionId oldest_running,
+ XLogRecPtr start_lsn,
bool need_full_snapshot,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index 8ca6050462..d060941e4f 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -57,7 +57,8 @@ extern bool TransactionIdIsActive(TransactionId xid);
extern TransactionId GetOldestNonRemovableTransactionId(Relation rel);
extern TransactionId GetOldestTransactionIdConsideredRunning(void);
extern TransactionId GetOldestActiveTransactionId(void);
-extern TransactionId GetOldestSafeDecodingTransactionId(bool catalogOnly);
+extern TransactionId GetOldestSafeDecodingTransactionId(bool catalogOnly,
+ TransactionId *oldestActiveXid_p);
extern void GetReplicationHorizons(TransactionId *xmin, TransactionId *catalog_xmin);
extern VirtualTransactionId *GetVirtualXIDsDelayingChkpt(int *nvxids, int type);
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-03-27 10:37 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-03-27 10:37 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
>
> With the PoC patch, we check ondisk.builder.is_there_running_xact in
> SnapBuildRestore(),
Yes, the PoC requires that the state of snapshot in the file must be read.
> but can we just check running->xcnt in
> SnapBuildFindSnapshot() to skip calling SnapBuildRestore()? That is,
> if builder->initial_xmin_horizon is valid (or
> builder->finding_start_point is true) and running->xcnt > 0, we skip
> the snapshot restore.
IIUC, it does not require modifications of API. It may be an advantage.
> However, I think there are still cases where we
> unnecessarily skip snapshot restores
>
> Probably, what we would like to avoid is, we compute
> initial_xmin_horizon and start to find the initial start point while
> there is a concurrently running transaction, and then jump to the
> consistent state by restoring the consistent snapshot before the
> concurrent transaction commits.
Yeah, information before concurrent txns are committed should not be used. I think
that's why SnapBuildWaitSnapshot() waits until listed transactions are finished.
> So we can ignore snapshot restores if
> (oldest XID among transactions running at the time of
> CreateInitDecodingContext()) >= (OldestRunningXID in
> xl_running_xacts).
>
> I've drafted this idea in the attached patch just for discussion.
Thanks for sharing the patch. At least I confirmed all tests and workload you
pointed out in [1] were passed. I will post here if I found other issues.
[1]: https://www.postgresql.org/message-id/CAD21AoDzLY9vRpo%2Bxb2qPtfn46ikiULPXDpT94sPyFH4GE8bYg%40mail.g...
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-04-16 03:56 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-04-16 03:56 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Mar 18, 2024 at 6:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Feb 19, 2024 at 2:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Feb 8, 2024 at 2:54 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Sat, Feb 3, 2024 at 12:12 PM Callahan, Drew <callaan@amazon.com> wrote:
> > > >
> > > > > I think we can prepare an isolation test of the above steps and
> > > > > include it in contrib/test_decoding/specs.
> > > >
> > > > Thanks for the suggestion. Included an isolation test for the
> > > > exact repro and a couple other tests for fully before and fully after.
> > > > These two tests don't add a whole lot of value since this
> > > > is already heavily tested elsewhere, so I'm good with removing.
> > > > However, thought it made sense to include from a perspective of
> > > > totality given the test name.
> > > >
> > > > > Another simple way to fix this is that we always set
> > > > > need_full_snapshot to true when slot creation. But it would then make
> > > > > the snapshot builder include non-catalog-change transactions too while
> > > > > finding the initial startpoint. Which is not necessary.
> > > >
> > > > Since we're not accumulating changes anyway and we never distribute snapshots
> > > > while inconsistent it wouldn't cause any memory pressure in the ReorderBuffer.
> > > > This seems pretty contained and wouldn't require the use of a global variable
> > > > and memory context callbacks which is a bit nicer from a readability standpoint.
> > >
> > > Sorry I missed this comment.
> > >
> > > I think it's a good point that this fix doesn't lead to any on-disk
> > > compatibility while not affecting existing users much, especially for
> > > back-patching. If we want to choose this approach for bank branches,
> > > we need to carefully consider the side impacts.
> > >
> > > > It's a little inefficient with memory and CPU, especially in the presence of
> > > > long running transactions, but likely would not be noticeable to most users.
> > >
> > > This matches my analysis.
> > >
> > > Here is the summary of several proposals we've discussed:
> > >
> > > a) Have CreateInitDecodingContext() always pass need_full_snapshot =
> > > true to AllocateSnapshotBuilder().
> > >
> > > This is the simplest approach and doesn't break the on-disk
> > > SnapBuildOnDisck compatibility. A downside is that it's a little
> > > inefficient with memory and CPU since it includes non-catalog change
> > > transactions too. Also, passing need_full_snapshot to
> > > CreateInitDecodingContext() will no longer have any meaning.
> > >
> > > b) Have snapbuild.c being able to handle multiple SnapBuildOnDisk versions.
> > >
> > > This bumps SNAPBUILD_VERSION and therefore breaks the on-disk
> > > SnapBuildOnDisk compatibility, but snapbuild.c can handle multiple
> > > versions. But since different branches are using different versions of
> > > SnapBuildOnDisk, it would probably be hard to have the SnapBuildOnDisk
> > > version that is consistent across all versions.
> > >
> > > c) Add a global variable, say in_create, to snapbuild.c
> > >
> > > it would also require adding a setter function for in_create. There
> > > are several ideas where to set/reset the flag. One idea is that we
> > > reset the flag in AllocateSnapshotBuilder() and set the flag anywhere
> > > before starting to find the start point, for example at the beginning
> > > of DecodingContextFindStartpoint(). It probably won't require a memory
> > > context callback to make sure to clear the flag. This idea doesn't
> > > have a downside from users and extensions perspects. But I'm slightly
> > > hesitant to add a global variable.
> > >
> > > What do you think? and any other ideas?
> >
> > I've drafted the idea (c) for discussion (for master and v16 for now).
> > I also liked the idea (a) but I'm concerned a bit about future impact.
> >
>
> I agree that (c) would be better than (a) as it avoids collecting
> non-catalog xacts in snapshot. However, I think we shouldn't avoid
> restoring the snapshot unless really required.
Pondering further, I came across the question; in what case we would
need to restore the snapshot and jump to the consistent state when
finding the initial start point?
When creating a slot, in ReplicationSlotReserveWal(), we determine the
restart_lsn and write a RUNNING_XACT record. This record would be the
first RUNNING_XACT record that the logical decoding decodes in most
cases. In SnapBuildFindSnapshot(), if the record satisfies (a) (i.e.
running->oldestRunningXid == running->nextXid), it can jump to the
consistent state. If not, it means there were running transactions at
that time of the RUNNING_XACT record being produced. Therefore, we
must not restore the snapshot and jump to the consistent state.
Because otherwise, we end up deciding the start point in the middle of
a transaction that started before the RUNNING_XACT record. Probably
the same is true for all subsequent snapbuild states.
I might be missing something but I could not find the case where we
can or want to restore the serialized snapshot when finding the
initial start point. If my analysis is correct, we can go with either
(a) or (c) I proposed before[1]. Between these two options, it also
could be an option that (a) is for backbranches for safety and (c) is
for master.
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoBH4NWdRcEjXpBFHSKuVO6wia-vHHHaKuEi-h4i4wbi8A%40mail.gma...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-20 06:19 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-06-20 06:19 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Apr 16, 2024 at 9:27 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Pondering further, I came across the question; in what case we would
> need to restore the snapshot and jump to the consistent state when
> finding the initial start point?
>
> When creating a slot, in ReplicationSlotReserveWal(), we determine the
> restart_lsn and write a RUNNING_XACT record. This record would be the
> first RUNNING_XACT record that the logical decoding decodes in most
> cases.
>
This probably won't be true on standby as we don't LOG RUNNING_XACT
record in that case.
> In SnapBuildFindSnapshot(), if the record satisfies (a) (i.e.
> running->oldestRunningXid == running->nextXid), it can jump to the
> consistent state. If not, it means there were running transactions at
> that time of the RUNNING_XACT record being produced. Therefore, we
> must not restore the snapshot and jump to the consistent state.
> Because otherwise, we end up deciding the start point in the middle of
> a transaction that started before the RUNNING_XACT record. Probably
> the same is true for all subsequent snapbuild states.
>
One thing to consider on this is that we can only restore the snapshot
if by that time some other get changes (in your case, step
"s2_get_changes_slot0" as in your draft patch) would have serialized
the consistent snapshot at that LSN. One could question if we can't
reach a consistent state at a particular LSN then why in the first
place the snapshot has been serialized at that LSN? The answer could
be that it is okay to use such a serialized snapshot after initial
snapshot creation because we know that the restart_lsn of a slot in
such cases would be a location where we won't see the data for partial
transactions. This is because, after the very first time (after the
initdecodingcontext), the restart_lsn would be set to a location after
we reach the consistent point.
> I might be missing something but I could not find the case where we
> can or want to restore the serialized snapshot when finding the
> initial start point. If my analysis is correct, we can go with either
> (a) or (c) I proposed before[1]. Between these two options, it also
> could be an option that (a) is for backbranches for safety and (c) is
> for master.
>
The approach (a) has a downside, it will lead to tracking more
transactions (non-catalog) than required without any benefit for the
user. Considering that is true, I wouldn't prefer that approach.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-21 06:46 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-06-21 06:46 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 20, 2024 at 3:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Apr 16, 2024 at 9:27 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Pondering further, I came across the question; in what case we would
> > need to restore the snapshot and jump to the consistent state when
> > finding the initial start point?
> >
> > When creating a slot, in ReplicationSlotReserveWal(), we determine the
> > restart_lsn and write a RUNNING_XACT record. This record would be the
> > first RUNNING_XACT record that the logical decoding decodes in most
> > cases.
> >
>
> This probably won't be true on standby as we don't LOG RUNNING_XACT
> record in that case.
Yes, on the standby we wait for the RUNNING_XACT record before
starting actual decoding.
>
> > In SnapBuildFindSnapshot(), if the record satisfies (a) (i.e.
> > running->oldestRunningXid == running->nextXid), it can jump to the
> > consistent state. If not, it means there were running transactions at
> > that time of the RUNNING_XACT record being produced. Therefore, we
> > must not restore the snapshot and jump to the consistent state.
> > Because otherwise, we end up deciding the start point in the middle of
> > a transaction that started before the RUNNING_XACT record. Probably
> > the same is true for all subsequent snapbuild states.
> >
>
> One thing to consider on this is that we can only restore the snapshot
> if by that time some other get changes (in your case, step
> "s2_get_changes_slot0" as in your draft patch) would have serialized
> the consistent snapshot at that LSN. One could question if we can't
> reach a consistent state at a particular LSN then why in the first
> place the snapshot has been serialized at that LSN? The answer could
> be that it is okay to use such a serialized snapshot after initial
> snapshot creation because we know that the restart_lsn of a slot in
> such cases would be a location where we won't see the data for partial
> transactions.
True. After the CreateInitDecodingContext() the restart_lsn is just an
arbitrary LSN in a sense; either the latest replay LSN or the current
insertion LSN. We need to find an LSN for the start point that
corresponds to that restart_lsn. So I think we need to find the start
point while disabling snapshot restores.
>
> > I might be missing something but I could not find the case where we
> > can or want to restore the serialized snapshot when finding the
> > initial start point. If my analysis is correct, we can go with either
> > (a) or (c) I proposed before[1]. Between these two options, it also
> > could be an option that (a) is for backbranches for safety and (c) is
> > for master.
> >
>
> The approach (a) has a downside, it will lead to tracking more
> transactions (non-catalog) than required without any benefit for the
> user. Considering that is true, I wouldn't prefer that approach.
Yes, it will lead to tracking non-catalog-change transactions as well.
If there are many subtransactions, the overhead could be noticeable.
But it happens only once when creating a slot.
Another variant of (a) is that we skip snapshot restores if the
initial_xmin_hirizon is a valid transaction id. The
initia_xmin_horizon is always set to a valida transaction id when
initializing the decoding context, e.g. during
CreateInitDecodingContext(). That way, we don't need to track
non-catalog-change transctions. A downside is that this approach
assumes that DecodingContextFindStartpoint() is called with the
decoding context created by CreateInitDecodingContxt(), which is true
in the core codes, but might not be true in third party extensions.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-24 03:54 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-06-24 03:54 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > The approach (a) has a downside, it will lead to tracking more
> > transactions (non-catalog) than required without any benefit for the
> > user. Considering that is true, I wouldn't prefer that approach.
>
> Yes, it will lead to tracking non-catalog-change transactions as well.
> If there are many subtransactions, the overhead could be noticeable.
> But it happens only once when creating a slot.
>
True, but it doesn't seem advisable to add such an overhead even
during create time without any concrete reason.
> Another variant of (a) is that we skip snapshot restores if the
> initial_xmin_hirizon is a valid transaction id. The
> initia_xmin_horizon is always set to a valida transaction id when
> initializing the decoding context, e.g. during
> CreateInitDecodingContext(). That way, we don't need to track
> non-catalog-change transctions. A downside is that this approach
> assumes that DecodingContextFindStartpoint() is called with the
> decoding context created by CreateInitDecodingContxt(), which is true
> in the core codes, but might not be true in third party extensions.
>
I think it is better to be explicit in this case rather than relying
on initia_xmin_horizon. So, we can store in_create/create_in_progress
flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
in back branches. I think changing SnapBuild means we have to update
SNAPBUILD_VERSION, right? Is that a good idea to do at this point of
time or shall we wait new branch to open and change it there? Anyway,
it would be a few days away and in the meantime, we can review and
keep the patches ready.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-24 05:01 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-06-24 05:01 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jun 24, 2024 at 12:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > The approach (a) has a downside, it will lead to tracking more
> > > transactions (non-catalog) than required without any benefit for the
> > > user. Considering that is true, I wouldn't prefer that approach.
> >
> > Yes, it will lead to tracking non-catalog-change transactions as well.
> > If there are many subtransactions, the overhead could be noticeable.
> > But it happens only once when creating a slot.
> >
>
> True, but it doesn't seem advisable to add such an overhead even
> during create time without any concrete reason.
>
> > Another variant of (a) is that we skip snapshot restores if the
> > initial_xmin_hirizon is a valid transaction id. The
> > initia_xmin_horizon is always set to a valida transaction id when
> > initializing the decoding context, e.g. during
> > CreateInitDecodingContext(). That way, we don't need to track
> > non-catalog-change transctions. A downside is that this approach
> > assumes that DecodingContextFindStartpoint() is called with the
> > decoding context created by CreateInitDecodingContxt(), which is true
> > in the core codes, but might not be true in third party extensions.
> >
>
> I think it is better to be explicit in this case rather than relying
> on initia_xmin_horizon. So, we can store in_create/create_in_progress
> flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
> in back branches.
I think we cannot access the flag in LogicalDecodingContext from
snapbuild.c at least in backbranches. I've discussed adding such a
flag in snapbuild.c as a global variable, but I'm slightly hesitant to
add a global variable besides InitialRunningXacts.
> I think changing SnapBuild means we have to update
> SNAPBUILD_VERSION, right? Is that a good idea to do at this point of
> time or shall we wait new branch to open and change it there? Anyway,
> it would be a few days away and in the meantime, we can review and
> keep the patches ready.
I think we should wait to add such changes that break on-disk
compatibility until a new branch opens. On HEAD, I think we can add a
new flag in SnapBuild and set it during say
DecodingContextFindStartpoint().
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-25 04:24 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-06-25 04:24 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jun 24, 2024 at 10:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jun 24, 2024 at 12:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > The approach (a) has a downside, it will lead to tracking more
> > > > transactions (non-catalog) than required without any benefit for the
> > > > user. Considering that is true, I wouldn't prefer that approach.
> > >
> > > Yes, it will lead to tracking non-catalog-change transactions as well.
> > > If there are many subtransactions, the overhead could be noticeable.
> > > But it happens only once when creating a slot.
> > >
> >
> > True, but it doesn't seem advisable to add such an overhead even
> > during create time without any concrete reason.
> >
> > > Another variant of (a) is that we skip snapshot restores if the
> > > initial_xmin_hirizon is a valid transaction id. The
> > > initia_xmin_horizon is always set to a valida transaction id when
> > > initializing the decoding context, e.g. during
> > > CreateInitDecodingContext(). That way, we don't need to track
> > > non-catalog-change transctions. A downside is that this approach
> > > assumes that DecodingContextFindStartpoint() is called with the
> > > decoding context created by CreateInitDecodingContxt(), which is true
> > > in the core codes, but might not be true in third party extensions.
> > >
> >
> > I think it is better to be explicit in this case rather than relying
> > on initia_xmin_horizon. So, we can store in_create/create_in_progress
> > flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
> > in back branches.
>
> I think we cannot access the flag in LogicalDecodingContext from
> snapbuild.c at least in backbranches. I've discussed adding such a
> flag in snapbuild.c as a global variable, but I'm slightly hesitant to
> add a global variable besides InitialRunningXacts.
>
I agree that adding a global variable is not advisable. Can we pass
the flag stored in LogicalDecodingContext to snapbuild.c? That might
not be elegant but I don't have any better ideas.
> > I think changing SnapBuild means we have to update
> > SNAPBUILD_VERSION, right? Is that a good idea to do at this point of
> > time or shall we wait new branch to open and change it there? Anyway,
> > it would be a few days away and in the meantime, we can review and
> > keep the patches ready.
>
> I think we should wait to add such changes that break on-disk
> compatibility until a new branch opens. On HEAD, I think we can add a
> new flag in SnapBuild and set it during say
> DecodingContextFindStartpoint().
>
Fair enough.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-25 08:26 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-06-25 08:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 25, 2024 at 1:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jun 24, 2024 at 10:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jun 24, 2024 at 12:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > The approach (a) has a downside, it will lead to tracking more
> > > > > transactions (non-catalog) than required without any benefit for the
> > > > > user. Considering that is true, I wouldn't prefer that approach.
> > > >
> > > > Yes, it will lead to tracking non-catalog-change transactions as well.
> > > > If there are many subtransactions, the overhead could be noticeable.
> > > > But it happens only once when creating a slot.
> > > >
> > >
> > > True, but it doesn't seem advisable to add such an overhead even
> > > during create time without any concrete reason.
> > >
> > > > Another variant of (a) is that we skip snapshot restores if the
> > > > initial_xmin_hirizon is a valid transaction id. The
> > > > initia_xmin_horizon is always set to a valida transaction id when
> > > > initializing the decoding context, e.g. during
> > > > CreateInitDecodingContext(). That way, we don't need to track
> > > > non-catalog-change transctions. A downside is that this approach
> > > > assumes that DecodingContextFindStartpoint() is called with the
> > > > decoding context created by CreateInitDecodingContxt(), which is true
> > > > in the core codes, but might not be true in third party extensions.
> > > >
> > >
> > > I think it is better to be explicit in this case rather than relying
> > > on initia_xmin_horizon. So, we can store in_create/create_in_progress
> > > flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
> > > in back branches.
> >
> > I think we cannot access the flag in LogicalDecodingContext from
> > snapbuild.c at least in backbranches. I've discussed adding such a
> > flag in snapbuild.c as a global variable, but I'm slightly hesitant to
> > add a global variable besides InitialRunningXacts.
> >
>
> I agree that adding a global variable is not advisable. Can we pass
> the flag stored in LogicalDecodingContext to snapbuild.c?
Ah, I found a good path: snapbuild->reorder->private_data (storing a
pointer to a LogicalDecodingContext). This assumes private_data always
stores a pointer to a LogicalDecodingContext but I think that's find
at least for backbranches.
I've attached the patch for this idea for PG16.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] 0001-Fix-possibility-of-logical-decoding-partial-of-trans.patch (11.5K, ../../CAD21AoDR3h78U0hxdzWPuPL11nvJCWYMB8h+QoOjd82ZmXjfgw@mail.gmail.com/2-0001-Fix-possibility-of-logical-decoding-partial-of-trans.patch)
download | inline diff:
From 8749a13ef54bbb38f4900413625331f1e9869c8c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Tue, 25 Jun 2024 17:20:47 +0900
Subject: [PATCH] Fix possibility of logical decoding partial of transaction
changes.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch-through:
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 81 +++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 58 +++++++++++++
src/backend/replication/logical/logical.c | 7 +-
src/backend/replication/logical/snapbuild.c | 18 +++--
src/include/replication/logical.h | 6 ++
6 files changed, 165 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..cb0040660b
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,81 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s0_init s0_begin s0_savepoint s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert_cat s0_commit s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_savepoint: SAVEPOINT sp0;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert_cat: INSERT INTO tbl VALUES (1);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..982fea5a6c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,58 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
+
+# The last decoding restarts from the NEW_CID record in the subtransaction.
+# While processing it, a same ReorderBufferChange entry would be associated
+# with both the top and the sub transaction, as the first entry. This breaks
+# an assumption in AssertTXNLsnOrder() which the first_lsn of entries must be
+# strictly higher than previous.
+permutation "s0_init" "s0_begin" "s0_savepoint" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert_cat" "s0_commit" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..f4ff00bead 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -151,6 +151,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -296,6 +297,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -437,7 +440,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -573,7 +576,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..1f088dd2f2 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1314,6 +1314,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1323,10 +1325,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while creating
+ * a slot since in this case the restart LSN is an arbitrary LSN and we
+ * need to find the start point to extract changes where we won't see
+ * the data for partial transactions. Also, we cannot use this method
+ * when slot needs a full snapshot for export or direct use, as that
+ * snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1391,8 +1395,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 5f49554ea0..6dc2f4f004 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -100,6 +100,12 @@ typedef struct LogicalDecodingContext
*/
bool twophase_opt_given;
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
+
/*
* State for writing output.
*/
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-06-26 09:17 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-06-26 09:17 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 25, 2024 at 1:56 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 25, 2024 at 1:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jun 24, 2024 at 10:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Jun 24, 2024 at 12:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > The approach (a) has a downside, it will lead to tracking more
> > > > > > transactions (non-catalog) than required without any benefit for the
> > > > > > user. Considering that is true, I wouldn't prefer that approach.
> > > > >
> > > > > Yes, it will lead to tracking non-catalog-change transactions as well.
> > > > > If there are many subtransactions, the overhead could be noticeable.
> > > > > But it happens only once when creating a slot.
> > > > >
> > > >
> > > > True, but it doesn't seem advisable to add such an overhead even
> > > > during create time without any concrete reason.
> > > >
> > > > > Another variant of (a) is that we skip snapshot restores if the
> > > > > initial_xmin_hirizon is a valid transaction id. The
> > > > > initia_xmin_horizon is always set to a valida transaction id when
> > > > > initializing the decoding context, e.g. during
> > > > > CreateInitDecodingContext(). That way, we don't need to track
> > > > > non-catalog-change transctions. A downside is that this approach
> > > > > assumes that DecodingContextFindStartpoint() is called with the
> > > > > decoding context created by CreateInitDecodingContxt(), which is true
> > > > > in the core codes, but might not be true in third party extensions.
> > > > >
> > > >
> > > > I think it is better to be explicit in this case rather than relying
> > > > on initia_xmin_horizon. So, we can store in_create/create_in_progress
> > > > flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
> > > > in back branches.
> > >
> > > I think we cannot access the flag in LogicalDecodingContext from
> > > snapbuild.c at least in backbranches. I've discussed adding such a
> > > flag in snapbuild.c as a global variable, but I'm slightly hesitant to
> > > add a global variable besides InitialRunningXacts.
> > >
> >
> > I agree that adding a global variable is not advisable. Can we pass
> > the flag stored in LogicalDecodingContext to snapbuild.c?
>
> Ah, I found a good path: snapbuild->reorder->private_data (storing a
> pointer to a LogicalDecodingContext). This assumes private_data always
> stores a pointer to a LogicalDecodingContext but I think that's find
> at least for backbranches.
>
+1. This approach looks good to me.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-05 14:52 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-05 14:52 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jun 26, 2024 at 6:17 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 25, 2024 at 1:56 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 25, 2024 at 1:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Jun 24, 2024 at 10:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Mon, Jun 24, 2024 at 12:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Fri, Jun 21, 2024 at 12:16 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > > >
> > > > > > > The approach (a) has a downside, it will lead to tracking more
> > > > > > > transactions (non-catalog) than required without any benefit for the
> > > > > > > user. Considering that is true, I wouldn't prefer that approach.
> > > > > >
> > > > > > Yes, it will lead to tracking non-catalog-change transactions as well.
> > > > > > If there are many subtransactions, the overhead could be noticeable.
> > > > > > But it happens only once when creating a slot.
> > > > > >
> > > > >
> > > > > True, but it doesn't seem advisable to add such an overhead even
> > > > > during create time without any concrete reason.
> > > > >
> > > > > > Another variant of (a) is that we skip snapshot restores if the
> > > > > > initial_xmin_hirizon is a valid transaction id. The
> > > > > > initia_xmin_horizon is always set to a valida transaction id when
> > > > > > initializing the decoding context, e.g. during
> > > > > > CreateInitDecodingContext(). That way, we don't need to track
> > > > > > non-catalog-change transctions. A downside is that this approach
> > > > > > assumes that DecodingContextFindStartpoint() is called with the
> > > > > > decoding context created by CreateInitDecodingContxt(), which is true
> > > > > > in the core codes, but might not be true in third party extensions.
> > > > > >
> > > > >
> > > > > I think it is better to be explicit in this case rather than relying
> > > > > on initia_xmin_horizon. So, we can store in_create/create_in_progress
> > > > > flag in the Snapbuild in HEAD and store it in LogicalDecodingContext
> > > > > in back branches.
> > > >
> > > > I think we cannot access the flag in LogicalDecodingContext from
> > > > snapbuild.c at least in backbranches. I've discussed adding such a
> > > > flag in snapbuild.c as a global variable, but I'm slightly hesitant to
> > > > add a global variable besides InitialRunningXacts.
> > > >
> > >
> > > I agree that adding a global variable is not advisable. Can we pass
> > > the flag stored in LogicalDecodingContext to snapbuild.c?
> >
> > Ah, I found a good path: snapbuild->reorder->private_data (storing a
> > pointer to a LogicalDecodingContext). This assumes private_data always
> > stores a pointer to a LogicalDecodingContext but I think that's find
> > at least for backbranches.
> >
>
> +1. This approach looks good to me.
>
I've attached updated patches for HEAD and pg17 for now (I will create
the patch for other backbranches).
In the patches, I used a different approach in between HEAD and
backbranches. On HEAD, we store a flag indicating whether or not we
should skip snapshot restores into the SnapBuild struct and set it
only while finding the start point. Therefore we have to bump
SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
above; store the flag in LogicalDecodingContext and set it when
creating the LogicalDecodingContext for a new logical slot. A possible
downside of the approach taken for backbranches is that we implicitly
require for users to use the same LogicalDecodingContext for both
initializing the context for a new slot and finding its start point.
IIUC it was not strictly required. This restriction would not be a
problem at least in the core, but I'm not sure if there are no
external extensions that create a logical slot in that way. This is
the main reason why I used a different approach on HEAD and
backbranches. Therefore, if it does not matter, I think we can use the
same approach on all branches, which is better in terms of
maintainability.
Also, I excluded the test case for the problem that Kuroda-san
reported[1] since the reported problem occurred due to the same cause
of the problem originally reported on this thread. The bug that
decodes only partial transactions could lead to various symptoms. IIUC
these test cases test the same behavior.
Regards,
[1] https://www.postgresql.org/message-id/TYCPR01MB1207719C811F580A8774C79B7F52A2%40TYCPR01MB12077.jpnpr...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] HEAD_v1-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (12.1K, ../../CAD21AoCS7Xfswo0v_jxZqJ97ndq=59s4Z2GVY8YjQZU-t6198w@mail.gmail.com/2-HEAD_v1-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 6b08c51515efa25aebac2adcc463a78e4db68bcd Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v1] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On HEAD, a flag indicating whether snapshot restores should be skipped
has been added to the SnapBuild struct. This flag is set only while
finding the start point, and SNAPBUILD_VERSION has been bumpded due to
changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 47 +++++++++++++++++
.../specs/skip_snapshot_restore.spec | 51 +++++++++++++++++++
src/backend/replication/logical/logical.c | 6 +++
src/backend/replication/logical/snapbuild.c | 32 +++++++++---
src/include/replication/snapbuild.h | 1 +
6 files changed, 133 insertions(+), 7 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..98f07545f0
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,47 @@
+unused step name: s0_insert_cat
+unused step name: s0_savepoint
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..a0b0814ec4
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,51 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..eb32bc9e14 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -650,6 +650,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
{
ReplicationSlot *slot = ctx->slot;
+ /* Let snapshot builder start to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, true);
+
/* Initialize from where to start reading WAL. */
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
@@ -683,6 +686,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
if (slot->data.two_phase)
slot->data.two_phase_at = ctx->reader->EndRecPtr;
SpinLockRelease(&slot->mutex);
+
+ /* Complete to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, false);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..7e018f905f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,9 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /* Indicates if we are finding the start point to extract changes */
+ bool finding_start_point;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -435,6 +438,15 @@ SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr)
return ptr < builder->start_decoding_at;
}
+/*
+ * Let the snapshot builder start finding the start point.
+ */
+void
+SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point)
+{
+ builder->finding_start_point = find_start_point;
+}
+
/*
* Increase refcount of a snapshot.
*
@@ -1327,10 +1339,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point since in this case the restart LSN could
+ * be in the middle of transactions we need to find the start point
+ * where we won't see the data for partial transactions. Also, we
+ * cannot use this method when a slot needs a full snapshot for export
+ * or direct use, as that snapshot will only contain catalog modifying
+ * transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1410,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->finding_start_point &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1600,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..d44af34ba0 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -76,6 +76,7 @@ extern SnapBuildState SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
--
2.39.3
[application/octet-stream] rel17_v1-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.7K, ../../CAD21AoCS7Xfswo0v_jxZqJ97ndq=59s4Z2GVY8YjQZU-t6198w@mail.gmail.com/3-rel17_v1-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 49e8db84b248f5c0726ff1391ff14b7884b26eef Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v1] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On HEAD, a flag indicating whether snapshot restores should be skipped
has been added to the SnapBuild struct. This flag is set only while
finding the start point, and SNAPBUILD_VERSION has been bumpded due to
changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 47 +++++++++++++++++
.../specs/skip_snapshot_restore.spec | 51 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 16 ++++--
src/include/replication/logical.h | 6 +++
6 files changed, 124 insertions(+), 6 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..98f07545f0
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,47 @@
+unused step name: s0_insert_cat
+unused step name: s0_savepoint
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..a0b0814ec4
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,51 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ DROP TABLE IF EXISTS user_cat;
+ CREATE TABLE tbl (val1 integer);
+ CREATE TABLE user_cat (val1 integer) WITH (user_catalog_table = true);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ DROP TABLE user_cat;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_insert_cat" { INSERT INTO tbl VALUES (1); }
+step "s0_savepoint" { SAVEPOINT sp0; }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 97a4d99c4e..31c8ec4910 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -152,6 +152,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -297,6 +298,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -438,7 +441,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -592,7 +595,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..8e38ff69d2 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1318,6 +1318,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1327,8 +1329,11 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
+ * snapshot to disk that we can use. Can't use this method while creating
+ * a slot since in this case the restart LSN could be in the middle of
+ * transactions we need to need to find the start point to extract
+ * changes where we won't see the data for partial transactions.
+ * Also, we cannot use this method when a slot needs a full snapshot
* for export or direct use, as that snapshot will only contain catalog
* modifying transactions.
*
@@ -1395,8 +1400,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index aff38e8d04..698aa6f02f 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -100,6 +100,12 @@ typedef struct LogicalDecodingContext
*/
bool twophase_opt_given;
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
+
/*
* State for writing output.
*/
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-07 23:56 Michael Paquier <michael@paquier.xyz>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Michael Paquier @ 2024-07-07 23:56 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, Jul 05, 2024 at 11:52:52PM +0900, Masahiko Sawada wrote:
> I've attached updated patches for HEAD and pg17 for now (I will create
> the patch for other backbranches).
>
> In the patches, I used a different approach in between HEAD and
> backbranches. On HEAD, we store a flag indicating whether or not we
> should skip snapshot restores into the SnapBuild struct and set it
> only while finding the start point. Therefore we have to bump
> SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> above; store the flag in LogicalDecodingContext and set it when
> creating the LogicalDecodingContext for a new logical slot. A possible
> downside of the approach taken for backbranches is that we implicitly
> require for users to use the same LogicalDecodingContext for both
> initializing the context for a new slot and finding its start point.
> IIUC it was not strictly required. This restriction would not be a
> problem at least in the core, but I'm not sure if there are no
> external extensions that create a logical slot in that way. This is
> the main reason why I used a different approach on HEAD and
> backbranches. Therefore, if it does not matter, I think we can use the
> same approach on all branches, which is better in terms of
> maintainability.
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,9 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /* Indicates if we are finding the start point to extract changes */
+ bool finding_start_point;
+
FYI, I think that it is still OK to bump SNAPBUILD_VERSION on
REL_17_STABLE. That will reduce by 1 year the time window required to
maintain the tweaks implemented for the versions in the back-branches.
So I'd suggest to do what the v17 version of the patch does for ~16,
and use the snapshot format changes in 17~.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZosrOu7RFnrrsuHL@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-08 03:01 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-08 03:01 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jul 8, 2024 at 8:56 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Jul 05, 2024 at 11:52:52PM +0900, Masahiko Sawada wrote:
> > I've attached updated patches for HEAD and pg17 for now (I will create
> > the patch for other backbranches).
> >
> > In the patches, I used a different approach in between HEAD and
> > backbranches. On HEAD, we store a flag indicating whether or not we
> > should skip snapshot restores into the SnapBuild struct and set it
> > only while finding the start point. Therefore we have to bump
> > SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> > above; store the flag in LogicalDecodingContext and set it when
> > creating the LogicalDecodingContext for a new logical slot. A possible
> > downside of the approach taken for backbranches is that we implicitly
> > require for users to use the same LogicalDecodingContext for both
> > initializing the context for a new slot and finding its start point.
> > IIUC it was not strictly required. This restriction would not be a
> > problem at least in the core, but I'm not sure if there are no
> > external extensions that create a logical slot in that way. This is
> > the main reason why I used a different approach on HEAD and
> > backbranches. Therefore, if it does not matter, I think we can use the
> > same approach on all branches, which is better in terms of
> > maintainability.
>
> --- a/src/backend/replication/logical/snapbuild.c
> +++ b/src/backend/replication/logical/snapbuild.c
> @@ -189,6 +189,9 @@ struct SnapBuild
> /* Indicates if we are building full snapshot or just catalog one. */
> bool building_full_snapshot;
>
> + /* Indicates if we are finding the start point to extract changes */
> + bool finding_start_point;
> +
>
> FYI, I think that it is still OK to bump SNAPBUILD_VERSION on
> REL_17_STABLE. That will reduce by 1 year the time window required to
> maintain the tweaks implemented for the versions in the back-branches.
> So I'd suggest to do what the v17 version of the patch does for ~16,
> and use the snapshot format changes in 17~.
Thank you for the comment.
Agreed. If we get consensus on this approach to go, I'll use the patch
changing the SnapBuild struct for 17 and HEAD.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* RE: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-08 09:01 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 40+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-07-08 09:01 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
Thanks for creating the patch!
> I've attached updated patches for HEAD and pg17 for now (I will create
> the patch for other backbranches).
>
> In the patches, I used a different approach in between HEAD and
> backbranches. On HEAD, we store a flag indicating whether or not we
> should skip snapshot restores into the SnapBuild struct and set it
> only while finding the start point. Therefore we have to bump
> SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> above; store the flag in LogicalDecodingContext and set it when
> creating the LogicalDecodingContext for a new logical slot. A possible
> downside of the approach taken for backbranches is that we implicitly
> require for users to use the same LogicalDecodingContext for both
> initializing the context for a new slot and finding its start point.
> IIUC it was not strictly required. This restriction would not be a
> problem at least in the core, but I'm not sure if there are no
> external extensions that create a logical slot in that way. This is
> the main reason why I used a different approach on HEAD and
> backbranches. Therefore, if it does not matter, I think we can use the
> same approach on all branches, which is better in terms of
> maintainability.
I want to confirm your point. You meant that you wanted to unify appraoches,
especially you wanted to store the flag in LogicalDecodingContext, rigth? I
briefly grepped github repos with "DecodingContextFindStartpoint", and cannot
find such pattern. E.g., [1]. But this point must be pointed by extension developers.
>
> Also, I excluded the test case for the problem that Kuroda-san
> reported[1] since the reported problem occurred due to the same cause
> of the problem originally reported on this thread. The bug that
> decodes only partial transactions could lead to various symptoms. IIUC
> these test cases test the same behavior.
Yes, I also think they are caused by the same root cause, so we can skip.
Comments for HEAD patch:
1.
You must modify test_decoding/meson.build. It was also missing in patches for
backbranches.
2.
The test code needs cleanup. E.g.,
- Steps "s0_insert_cat" and "s0_savepoint" is not used
- Table user_cat is not used
It was also missing in patches for backbranches.
3.
Just to confirm - In SnapshotRestore(), the added attribute is not restored
from the disk. This is intentional because 1) it has been set to false when it
was serilizing to disk and 2) the destination (SnapBuild *builder) is initialized
by palloc0() so it was set to false. Is it right?
Comments for backblanches patch:
1.
According to wikipage [2], new attribute must be at the end of struct.
[1]: https://github.com/cybertec-postgresql/pg_squeeze/blob/master/worker.c#L1614
[2]: https://wiki.postgresql.org/wiki/Committing_checklist
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-08 14:02 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-08 14:02 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jul 8, 2024 at 6:01 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Sawada-san,
>
> Thanks for creating the patch!
>
> > I've attached updated patches for HEAD and pg17 for now (I will create
> > the patch for other backbranches).
> >
> > In the patches, I used a different approach in between HEAD and
> > backbranches. On HEAD, we store a flag indicating whether or not we
> > should skip snapshot restores into the SnapBuild struct and set it
> > only while finding the start point. Therefore we have to bump
> > SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> > above; store the flag in LogicalDecodingContext and set it when
> > creating the LogicalDecodingContext for a new logical slot. A possible
> > downside of the approach taken for backbranches is that we implicitly
> > require for users to use the same LogicalDecodingContext for both
> > initializing the context for a new slot and finding its start point.
> > IIUC it was not strictly required. This restriction would not be a
> > problem at least in the core, but I'm not sure if there are no
> > external extensions that create a logical slot in that way. This is
> > the main reason why I used a different approach on HEAD and
> > backbranches. Therefore, if it does not matter, I think we can use the
> > same approach on all branches, which is better in terms of
> > maintainability.
>
> I want to confirm your point. You meant that you wanted to unify appraoches,
> especially you wanted to store the flag in LogicalDecodingContext, rigth?
Yes. Ideally I'd like to use the same approach in all branches
regardless of how to fix it for better maintainability.
> I
> briefly grepped github repos with "DecodingContextFindStartpoint", and cannot
> find such pattern. E.g., [1]. But this point must be pointed by extension developers.
Thank you for searching on github. It will also affect the future
extension developments so I'm slightly hesitant to do that.
> >
> > Also, I excluded the test case for the problem that Kuroda-san
> > reported[1] since the reported problem occurred due to the same cause
> > of the problem originally reported on this thread. The bug that
> > decodes only partial transactions could lead to various symptoms. IIUC
> > these test cases test the same behavior.
>
> Yes, I also think they are caused by the same root cause, so we can skip.
>
> Comments for HEAD patch:
> 1.
> You must modify test_decoding/meson.build. It was also missing in patches for
> backbranches.
>
> 2.
> The test code needs cleanup. E.g.,
> - Steps "s0_insert_cat" and "s0_savepoint" is not used
> - Table user_cat is not used
> It was also missing in patches for backbranches.
Will fix the above points in the next version patch.
>
> 3.
> Just to confirm - In SnapshotRestore(), the added attribute is not restored
> from the disk. This is intentional because 1) it has been set to false when it
> was serilizing to disk and 2) the destination (SnapBuild *builder) is initialized
> by palloc0() so it was set to false. Is it right?
Right.
> Comments for backblanches patch:
>
> 1.
> According to wikipage [2], new attribute must be at the end of struct.
Good point. Will fix.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-09 05:29 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-09 05:29 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jul 8, 2024 at 11:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jul 8, 2024 at 6:01 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Sawada-san,
> >
> > Thanks for creating the patch!
> >
> > > I've attached updated patches for HEAD and pg17 for now (I will create
> > > the patch for other backbranches).
> > >
> > > In the patches, I used a different approach in between HEAD and
> > > backbranches. On HEAD, we store a flag indicating whether or not we
> > > should skip snapshot restores into the SnapBuild struct and set it
> > > only while finding the start point. Therefore we have to bump
> > > SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> > > above; store the flag in LogicalDecodingContext and set it when
> > > creating the LogicalDecodingContext for a new logical slot. A possible
> > > downside of the approach taken for backbranches is that we implicitly
> > > require for users to use the same LogicalDecodingContext for both
> > > initializing the context for a new slot and finding its start point.
> > > IIUC it was not strictly required. This restriction would not be a
> > > problem at least in the core, but I'm not sure if there are no
> > > external extensions that create a logical slot in that way. This is
> > > the main reason why I used a different approach on HEAD and
> > > backbranches. Therefore, if it does not matter, I think we can use the
> > > same approach on all branches, which is better in terms of
> > > maintainability.
> >
> > I want to confirm your point. You meant that you wanted to unify appraoches,
> > especially you wanted to store the flag in LogicalDecodingContext, rigth?
>
> Yes. Ideally I'd like to use the same approach in all branches
> regardless of how to fix it for better maintainability.
>
> > I
> > briefly grepped github repos with "DecodingContextFindStartpoint", and cannot
> > find such pattern. E.g., [1]. But this point must be pointed by extension developers.
>
> Thank you for searching on github. It will also affect the future
> extension developments so I'm slightly hesitant to do that.
>
> > >
> > > Also, I excluded the test case for the problem that Kuroda-san
> > > reported[1] since the reported problem occurred due to the same cause
> > > of the problem originally reported on this thread. The bug that
> > > decodes only partial transactions could lead to various symptoms. IIUC
> > > these test cases test the same behavior.
> >
> > Yes, I also think they are caused by the same root cause, so we can skip.
> >
> > Comments for HEAD patch:
> > 1.
> > You must modify test_decoding/meson.build. It was also missing in patches for
> > backbranches.
> >
> > 2.
> > The test code needs cleanup. E.g.,
> > - Steps "s0_insert_cat" and "s0_savepoint" is not used
> > - Table user_cat is not used
> > It was also missing in patches for backbranches.
>
> Will fix the above points in the next version patch.
>
> >
> > 3.
> > Just to confirm - In SnapshotRestore(), the added attribute is not restored
> > from the disk. This is intentional because 1) it has been set to false when it
> > was serilizing to disk and 2) the destination (SnapBuild *builder) is initialized
> > by palloc0() so it was set to false. Is it right?
>
> Right.
>
> > Comments for backblanches patch:
> >
> > 1.
> > According to wikipage [2], new attribute must be at the end of struct.
>
> Good point. Will fix.
>
I've attached the new version patches for all branches.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] REL14_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.7K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/2-REL14_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 8c89a99ca567d6ed6c6a12889105a23092c39b38 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 2 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 20 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 4553252d75..82ba3f7df1 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,7 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot catalog_change_snapshot
+ twophase_snapshot catalog_change_snapshot skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index f7d1491907..2f7b2c85d9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -286,6 +287,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -421,7 +424,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -528,7 +531,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 4f8c00f717..eaf5ddb98b 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1296,6 +1296,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1305,10 +1307,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1373,8 +1378,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 25490f221f..34d4cd5922 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -99,6 +99,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] master_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (12.3K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/3-master_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 6ce61c863e8a6aab530f56ac984d6adfc3c0442a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 6 +++
src/backend/replication/logical/snapbuild.c | 32 ++++++++++---
src/include/replication/snapbuild.h | 1 +
7 files changed, 127 insertions(+), 7 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..eb32bc9e14 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -650,6 +650,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
{
ReplicationSlot *slot = ctx->slot;
+ /* Let snapshot builder start to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, true);
+
/* Initialize from where to start reading WAL. */
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
@@ -683,6 +686,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
if (slot->data.two_phase)
slot->data.two_phase_at = ctx->reader->EndRecPtr;
SpinLockRelease(&slot->mutex);
+
+ /* Complete to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, false);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..4591df9c89 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,9 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /* Indicates if we are finding the start point to extract changes */
+ bool finding_start_point;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -435,6 +438,15 @@ SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr)
return ptr < builder->start_decoding_at;
}
+/*
+ * Let the snapshot builder start finding the start point.
+ */
+void
+SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point)
+{
+ builder->finding_start_point = find_start_point;
+}
+
/*
* Increase refcount of a snapshot.
*
@@ -1327,10 +1339,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1410,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->finding_start_point &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1600,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..d44af34ba0 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -76,6 +76,7 @@ extern SnapBuildState SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
--
2.39.3
[application/octet-stream] REL16_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (12.0K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/4-REL16_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 4aa7dcd506344c7790c84222bc71e9fb5fb5afc8 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 16 +++++--
src/include/replication/logical.h | 6 +++
7 files changed, 118 insertions(+), 6 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 7b05cc25a3..2dd3ede41b 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..6894d3acbc 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -151,6 +151,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -296,6 +297,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -437,7 +440,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -573,7 +576,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..fb24425d4e 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1314,6 +1314,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1323,8 +1325,11 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
+ * snapshot to disk that we can use. Can't use this method while creating
+ * a slot since in this case the restart LSN could be in the middle of
+ * transactions we need to need to find the start point to extract
+ * changes where we won't see the data for partial transactions.
+ * Also, we cannot use this method when a slot needs a full snapshot
* for export or direct use, as that snapshot will only contain catalog
* modifying transactions.
*
@@ -1391,8 +1396,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 5f49554ea0..c29c6acdd0 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL15_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.7K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/5-REL15_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 8ce751f6c964575286e69736797538b0c96505d7 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 20 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 119 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 466f30c22d..48ae02ca38 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -287,6 +288,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -422,7 +425,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -535,7 +538,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 39819947c1..fb13825951 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1305,6 +1305,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1314,10 +1316,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1382,8 +1387,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index edadacd589..6f4044bfda 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL17_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (12.3K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/6-REL17_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 0c7419cdca04b55dde7b519a5dff26dc5720b43d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 6 +++
src/backend/replication/logical/snapbuild.c | 32 ++++++++++---
src/include/replication/snapbuild.h | 1 +
7 files changed, 127 insertions(+), 7 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 97a4d99c4e..116559433e 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -650,6 +650,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
{
ReplicationSlot *slot = ctx->slot;
+ /* Let snapshot builder start to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, true);
+
/* Initialize from where to start reading WAL. */
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
@@ -683,6 +686,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
if (slot->data.two_phase)
slot->data.two_phase_at = ctx->reader->EndRecPtr;
SpinLockRelease(&slot->mutex);
+
+ /* Complete to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, false);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..4591df9c89 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,9 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /* Indicates if we are finding the start point to extract changes */
+ bool finding_start_point;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -435,6 +438,15 @@ SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr)
return ptr < builder->start_decoding_at;
}
+/*
+ * Let the snapshot builder start finding the start point.
+ */
+void
+SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point)
+{
+ builder->finding_start_point = find_start_point;
+}
+
/*
* Increase refcount of a snapshot.
*
@@ -1327,10 +1339,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1410,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->finding_start_point &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1600,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..d44af34ba0 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -76,6 +76,7 @@ extern SnapBuildState SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
--
2.39.3
[application/octet-stream] REL12_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.7K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/7-REL12_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From e6f16780e1fabb51e8f3b861e05f82cbdcb0f78b Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 20 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 119 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8b79d67001..e6423b955b 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -123,6 +123,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogPageReadCB read_page,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -201,6 +202,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -329,7 +332,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
read_page, prepare_write, do_write,
update_progress);
@@ -428,7 +431,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, read_page, prepare_write,
+ fast_forward, false, read_page, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index bdfd1f0228..2b86e4c8e6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1312,6 +1312,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1321,10 +1323,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1389,8 +1394,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 718080d54a..21923172cf 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -90,6 +90,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL13_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.7K, ../../CAD21AoBV7odVHcTsXS2VNUKqw3J0AB_jaRNhL0UuXJ7=kOkLFw@mail.gmail.com/8-REL13_v2-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From b8dbe20781d3f18a70884f2c663b737e56773f10 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v2] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct. This flag is set only
while finding the start point, and SNAPBUILD_VERSION has been bumpded
due to changes in the SnapBuild struct.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility. The flag is set only when
initializing the LogicalDecodingContext. Therefore, the context used
for finding the start point must be the same as the one used for
initializing the slot, which was not strictly requried before but is
not a practical issue.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 20 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 119 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 6b9f24e52d..70bd983dc2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -120,6 +120,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -198,6 +199,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -327,7 +330,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -429,7 +432,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 31caad8855..1177b9c293 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1307,6 +1307,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1316,10 +1318,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while
+ * finding the start point for decoding changes. Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions. Also, we cannot use this method when a slot needs a
+ * full snapshot for export or direct use, as that snapshot will only
+ * contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1384,8 +1389,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 14d71fd136..178baf1f95 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -89,6 +89,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-09 09:27 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-07-09 09:27 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jul 9, 2024 at 11:00 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jul 8, 2024 at 11:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jul 8, 2024 at 6:01 PM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > Dear Sawada-san,
> > >
> > > Thanks for creating the patch!
> > >
> > > > I've attached updated patches for HEAD and pg17 for now (I will create
> > > > the patch for other backbranches).
> > > >
> > > > In the patches, I used a different approach in between HEAD and
> > > > backbranches. On HEAD, we store a flag indicating whether or not we
> > > > should skip snapshot restores into the SnapBuild struct and set it
> > > > only while finding the start point. Therefore we have to bump
> > > > SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> > > > above; store the flag in LogicalDecodingContext and set it when
> > > > creating the LogicalDecodingContext for a new logical slot. A possible
> > > > downside of the approach taken for backbranches is that we implicitly
> > > > require for users to use the same LogicalDecodingContext for both
> > > > initializing the context for a new slot and finding its start point.
> > > > IIUC it was not strictly required. This restriction would not be a
> > > > problem at least in the core, but I'm not sure if there are no
> > > > external extensions that create a logical slot in that way. This is
> > > > the main reason why I used a different approach on HEAD and
> > > > backbranches. Therefore, if it does not matter, I think we can use the
> > > > same approach on all branches, which is better in terms of
> > > > maintainability.
> > >
> > > I want to confirm your point. You meant that you wanted to unify appraoches,
> > > especially you wanted to store the flag in LogicalDecodingContext, rigth?
> >
> > Yes. Ideally I'd like to use the same approach in all branches
> > regardless of how to fix it for better maintainability.
> >
The difference is minor so using slightly different approaches should
be okay. In the ideal case, I agree that using the same approach makes
sense but for future extendability, it is better to keep this new
variable in SnapBuild at least in HEAD and PG17.
>
> I've attached the new version patches for all branches.
>
Few comments:
1.
@@ -650,6 +650,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
{
ReplicationSlot *slot = ctx->slot;
+ /* Let snapshot builder start to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, true);
+
/* Initialize from where to start reading WAL. */
XLogBeginRead(ctx->reader, slot->data.restart_lsn);
@@ -683,6 +686,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
if (slot->data.two_phase)
slot->data.two_phase_at = ctx->reader->EndRecPtr;
SpinLockRelease(&slot->mutex);
+
+ /* Complete to find the start point */
+ SnapBuildSetFindStartPoint(ctx->snapshot_builder, false);
I wonder why you didn't choose to set this variable in
AllocateSnapshotBuilder()? If we do so, then we may not need set/reset
in DecodingContextFindStartpoint(). The one advantage of using
set/reset for the minimal time as done here is that we can avoid any
impact of this new variable but I still feel setting it in
AllocateSnapshotBuilder() seems more natural.
2.
Since in this case
+ * the restart LSN could be in the middle of transactions we need to
+ * find the start point where we won't see the data for partial
+ * transactions.
There is a connecting word missing between *transactions* and *we*.
Can we use a slightly different wording like: "Can't use this method
while finding the start point for decoding changes as the restart LSN
would be an arbitrary LSN but we need to find the start point to
extract changes where we won't see the data for partial
transactions."?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-10 01:15 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-10 01:15 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jul 9, 2024 at 6:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 9, 2024 at 11:00 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jul 8, 2024 at 11:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Jul 8, 2024 at 6:01 PM Hayato Kuroda (Fujitsu)
> > > <kuroda.hayato@fujitsu.com> wrote:
> > > >
> > > > Dear Sawada-san,
> > > >
> > > > Thanks for creating the patch!
> > > >
> > > > > I've attached updated patches for HEAD and pg17 for now (I will create
> > > > > the patch for other backbranches).
> > > > >
> > > > > In the patches, I used a different approach in between HEAD and
> > > > > backbranches. On HEAD, we store a flag indicating whether or not we
> > > > > should skip snapshot restores into the SnapBuild struct and set it
> > > > > only while finding the start point. Therefore we have to bump
> > > > > SNAPBUILD_VERSION. On backbranches, I used the approach we discussed
> > > > > above; store the flag in LogicalDecodingContext and set it when
> > > > > creating the LogicalDecodingContext for a new logical slot. A possible
> > > > > downside of the approach taken for backbranches is that we implicitly
> > > > > require for users to use the same LogicalDecodingContext for both
> > > > > initializing the context for a new slot and finding its start point.
> > > > > IIUC it was not strictly required. This restriction would not be a
> > > > > problem at least in the core, but I'm not sure if there are no
> > > > > external extensions that create a logical slot in that way. This is
> > > > > the main reason why I used a different approach on HEAD and
> > > > > backbranches. Therefore, if it does not matter, I think we can use the
> > > > > same approach on all branches, which is better in terms of
> > > > > maintainability.
> > > >
> > > > I want to confirm your point. You meant that you wanted to unify appraoches,
> > > > especially you wanted to store the flag in LogicalDecodingContext, rigth?
> > >
> > > Yes. Ideally I'd like to use the same approach in all branches
> > > regardless of how to fix it for better maintainability.
> > >
>
> The difference is minor so using slightly different approaches should
> be okay. In the ideal case, I agree that using the same approach makes
> sense but for future extendability, it is better to keep this new
> variable in SnapBuild at least in HEAD and PG17.
>
> >
> > I've attached the new version patches for all branches.
> >
>
> Few comments:
> 1.
> @@ -650,6 +650,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
> {
> ReplicationSlot *slot = ctx->slot;
>
> + /* Let snapshot builder start to find the start point */
> + SnapBuildSetFindStartPoint(ctx->snapshot_builder, true);
> +
> /* Initialize from where to start reading WAL. */
> XLogBeginRead(ctx->reader, slot->data.restart_lsn);
>
> @@ -683,6 +686,9 @@ DecodingContextFindStartpoint(LogicalDecodingContext *ctx)
> if (slot->data.two_phase)
> slot->data.two_phase_at = ctx->reader->EndRecPtr;
> SpinLockRelease(&slot->mutex);
> +
> + /* Complete to find the start point */
> + SnapBuildSetFindStartPoint(ctx->snapshot_builder, false);
>
> I wonder why you didn't choose to set this variable in
> AllocateSnapshotBuilder()? If we do so, then we may not need set/reset
> in DecodingContextFindStartpoint(). The one advantage of using
> set/reset for the minimal time as done here is that we can avoid any
> impact of this new variable but I still feel setting it in
> AllocateSnapshotBuilder() seems more natural.
Initially what I thought was; if we set the flag in
AllocateSnapshotBuilder(), we need to pass true through like
CreateInitDecodingContext() -> StartupDecodingContext() ->
AllocateSnapshotBuilder(), meaning that the only
LogicalDecodingContext created via CreateInitDecodingContext() can be
used in DecodingContextFindStartpoint(). IOW if we use the
LogicalDecodingContext created via CreateDecodingContext() (i.e.,
setting the flag as false) in DecodingContextFindStartpoint() we would
end up having the same problem. IIUC we haven't had such a usage
restriction before. But thinking on this approach further, probably
the same is true for initial_xmin_horizon. The field is set only when
the LogicalDecodingContext is created via CreateInitDecodingContext(),
and is used only while finding the start point. So I'm fine with
setting the flag in AllocateSnapshotBuilder(). If we go this approach,
I think we should check if the flag is set before finding the start
point.
>
> 2.
> Since in this case
> + * the restart LSN could be in the middle of transactions we need to
> + * find the start point where we won't see the data for partial
> + * transactions.
>
> There is a connecting word missing between *transactions* and *we*.
> Can we use a slightly different wording like: "Can't use this method
> while finding the start point for decoding changes as the restart LSN
> would be an arbitrary LSN but we need to find the start point to
> extract changes where we won't see the data for partial
> transactions."?
Looks good. Will fix.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-10 02:21 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-10 02:21 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jul 10, 2024 at 10:15 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> If we go this approach,
> I think we should check if the flag is set before finding the start
> point.
Checking if the flag is set in DecodingContextFindStartpoint()
requires introducing a new function in snapbuild.c and the function
would be used only in an Assertion. So the attached updated patch
doesn't introduce such a function. If it's really worth it, I can add
it, but I think we can live without that.
Overall I think the patches are in good shape, so I'm going to push
them tomorrow, barring any objections and further comments.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] master_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (13.4K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/2-master_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From d0b4fcb8f30bfba3fb08ac2c8e6f9a0e25ad858b Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 +--
src/backend/replication/logical/snapbuild.c | 29 +++++++++---
src/include/replication/snapbuild.h | 2 +
7 files changed, 123 insertions(+), 10 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..f8ef5d56d2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -152,6 +152,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -212,7 +213,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ need_full_snapshot, in_create, slot->data.two_phase_at);
ctx->reorder->private_data = ctx;
@@ -438,7 +439,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -592,7 +593,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..bd2e53790a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,14 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * Indicates if we are using the snapshot builder for the initial creation
+ * of a logical replication slot. If it's true, the start point for
+ * decoding changes is not determined yet. So we skip snapshot restores to
+ * properly find the start point. See SnapBuildFindSnapshot() for details.
+ */
+ bool in_slot_creation;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -317,6 +325,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at)
{
MemoryContext context;
@@ -347,6 +356,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->initial_xmin_horizon = xmin_horizon;
builder->start_decoding_at = start_lsn;
+ builder->in_slot_creation = in_slot_creation;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
@@ -1327,10 +1337,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1407,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->in_slot_creation &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1597,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..9e6422564d 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -62,6 +62,7 @@ extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
@@ -76,6 +77,7 @@ extern SnapBuildState SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
--
2.39.3
[application/octet-stream] REL17_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (13.4K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/3-REL17_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 407864abda01ab9d98c7a28796c6d8ad15dc015d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 +--
src/backend/replication/logical/snapbuild.c | 29 +++++++++---
src/include/replication/snapbuild.h | 2 +
7 files changed, 123 insertions(+), 10 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..f8ef5d56d2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -152,6 +152,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -212,7 +213,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ need_full_snapshot, in_create, slot->data.two_phase_at);
ctx->reorder->private_data = ctx;
@@ -438,7 +439,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -592,7 +593,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..bd2e53790a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,14 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * Indicates if we are using the snapshot builder for the initial creation
+ * of a logical replication slot. If it's true, the start point for
+ * decoding changes is not determined yet. So we skip snapshot restores to
+ * properly find the start point. See SnapBuildFindSnapshot() for details.
+ */
+ bool in_slot_creation;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -317,6 +325,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at)
{
MemoryContext context;
@@ -347,6 +356,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->initial_xmin_horizon = xmin_horizon;
builder->start_decoding_at = start_lsn;
+ builder->in_slot_creation = in_slot_creation;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
@@ -1327,10 +1337,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1407,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->in_slot_creation &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1597,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..9e6422564d 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -62,6 +62,7 @@ extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
@@ -76,6 +77,7 @@ extern SnapBuildState SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool find_start_point);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
--
2.39.3
[application/octet-stream] REL15_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.4K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/4-REL15_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From f19037847118572c754583151402125071872e81 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 466f30c22d..48ae02ca38 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -287,6 +288,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -422,7 +425,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -535,7 +538,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 39819947c1..00edbfdbaf 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1305,6 +1305,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1314,10 +1316,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1382,8 +1386,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index edadacd589..6f4044bfda 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL14_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/5-REL14_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 2c9ab650c01a1ae7f165ac11a5e078836cbd4fab Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 2 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 117 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 4553252d75..82ba3f7df1 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,7 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot catalog_change_snapshot
+ twophase_snapshot catalog_change_snapshot skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index f7d1491907..2f7b2c85d9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -286,6 +287,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -421,7 +424,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -528,7 +531,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 4f8c00f717..8397394f7c 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1296,6 +1296,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1305,10 +1307,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1373,8 +1377,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 25490f221f..34d4cd5922 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -99,6 +99,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL16_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.9K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/6-REL16_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From b890becf206981da4563dfd5d1cb6635027b16ec Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
7 files changed, 119 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 7b05cc25a3..2dd3ede41b 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..6894d3acbc 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -151,6 +151,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -296,6 +297,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -437,7 +440,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -573,7 +576,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..e14140f749 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1314,6 +1314,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1323,10 +1325,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1391,8 +1395,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 5f49554ea0..c29c6acdd0 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL13_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/7-REL13_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From a3f0747c6c3f79f1ed3d28c7c2df9a02b47df6ff Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 6b9f24e52d..70bd983dc2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -120,6 +120,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -198,6 +199,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -327,7 +330,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -429,7 +432,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 31caad8855..089893bb65 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1307,6 +1307,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1316,10 +1318,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1384,8 +1388,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 14d71fd136..178baf1f95 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -89,6 +89,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL12_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoBm5_FzZFoFcfPf8c2G91hE4soQOiy_QAM1-mwv-UryXw@mail.gmail.com/8-REL12_v3-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 7967da59173d1e25e0d6e0c2e68ade2679b2961f Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v3] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8b79d67001..e6423b955b 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -123,6 +123,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogPageReadCB read_page,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -201,6 +202,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -329,7 +332,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
read_page, prepare_write, do_write,
update_progress);
@@ -428,7 +431,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, read_page, prepare_write,
+ fast_forward, false, read_page, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index bdfd1f0228..1941ee2f61 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1312,6 +1312,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1321,10 +1323,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1389,8 +1393,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 718080d54a..21923172cf 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -90,6 +90,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the initial
+ * creation of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-10 03:43 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Amit Kapila @ 2024-07-10 03:43 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jul 10, 2024 at 7:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Overall I think the patches are in good shape, so I'm going to push
> them tomorrow, barring any objections and further comments.
>
Agreed. Below are a few minor comments that you might want to consider:
1.
@@ -76,6 +77,7 @@ extern SnapBuildState
SnapBuildCurrentState(SnapBuild *builder);
extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool
find_start_point);
This is not required in the latest version.
2.
+ /*
+ * Indicates if we are using the snapshot builder for the initial creation
+ * of a logical replication slot.
The word 'initial' in the above comment is not required. If you
consider this then a similar wording change is required in lower
branches as well.
3.
HEAD and v17
---------------------
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * finding the start point.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->in_slot_creation &&
V16 and below
---------------------
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and neither building full snapshot nor while
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
Isn't it better to use the same comment in both places?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-10 05:30 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-10 05:30 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jul 10, 2024 at 12:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jul 10, 2024 at 7:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Overall I think the patches are in good shape, so I'm going to push
> > them tomorrow, barring any objections and further comments.
> >
>
> Agreed. Below are a few minor comments that you might want to consider:
>
> 1.
> @@ -76,6 +77,7 @@ extern SnapBuildState
> SnapBuildCurrentState(SnapBuild *builder);
> extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
>
> extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
> +extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool
> find_start_point);
>
> This is not required in the latest version.
>
> 2.
> + /*
> + * Indicates if we are using the snapshot builder for the initial creation
> + * of a logical replication slot.
>
> The word 'initial' in the above comment is not required. If you
> consider this then a similar wording change is required in lower
> branches as well.
>
> 3.
> HEAD and v17
> ---------------------
> - /* b) valid on disk state and not building full snapshot */
> +
> + /*
> + * b) valid on disk state and while neither building full snapshot nor
> + * finding the start point.
> + */
> else if (!builder->building_full_snapshot &&
> + !builder->in_slot_creation &&
>
> V16 and below
> ---------------------
> - /* b) valid on disk state and not building full snapshot */
> +
> + /*
> + * b) valid on disk state and neither building full snapshot nor while
> + * creating a slot.
> + */
> else if (!builder->building_full_snapshot &&
> + !ctx->in_create &&
>
> Isn't it better to use the same comment in both places?
Thank you for reviewing the patches! I agreed with all the points.
I've attached the updated patches.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] REL17_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (13.0K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/2-REL17_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 0708bd1dc713a15991a346be725a7b5cc9e1a7e1 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 +--
src/backend/replication/logical/snapbuild.c | 29 +++++++++---
src/include/replication/snapbuild.h | 1 +
7 files changed, 122 insertions(+), 10 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..f8ef5d56d2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -152,6 +152,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -212,7 +213,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ need_full_snapshot, in_create, slot->data.two_phase_at);
ctx->reorder->private_data = ctx;
@@ -438,7 +439,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -592,7 +593,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..ae676145e6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,14 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * Indicates if we are using the snapshot builder for the creation of a
+ * logical replication slot. If it's true, the start point for decoding
+ * changes is not determined yet. So we skip snapshot restores to properly
+ * find the start point. See SnapBuildFindSnapshot() for details.
+ */
+ bool in_slot_creation;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -317,6 +325,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at)
{
MemoryContext context;
@@ -347,6 +356,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->initial_xmin_horizon = xmin_horizon;
builder->start_decoding_at = start_lsn;
+ builder->in_slot_creation = in_slot_creation;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
@@ -1327,10 +1337,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1407,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->in_slot_creation &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1597,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..caa5113ff8 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -62,6 +62,7 @@ extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
--
2.39.3
[application/octet-stream] REL15_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.4K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/3-REL15_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From ac53c29485c57d31dd811fcecf802445a2c0ea8c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 466f30c22d..48ae02ca38 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -287,6 +288,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -422,7 +425,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -535,7 +538,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 39819947c1..cc1f2a9f15 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1305,6 +1305,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1314,10 +1316,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1382,8 +1386,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index edadacd589..030ce1eef5 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the creation
+ * of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL14_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/4-REL14_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 206a3d7186fcb95417a8db3ad96ea80f234b262a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 2 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 117 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 4553252d75..82ba3f7df1 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,7 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot catalog_change_snapshot
+ twophase_snapshot catalog_change_snapshot skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index f7d1491907..2f7b2c85d9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -148,6 +148,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -286,6 +287,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -421,7 +424,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -528,7 +531,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 4f8c00f717..9d8700147c 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1296,6 +1296,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1305,10 +1307,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1373,8 +1377,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 25490f221f..daf458cf98 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -99,6 +99,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the creation
+ * of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL16_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.8K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/5-REL16_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 6ee421c6f0d52227722197ea343b91e3928c48f1 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
7 files changed, 119 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 7b05cc25a3..2dd3ede41b 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 41243d0187..6894d3acbc 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -151,6 +151,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -296,6 +297,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -437,7 +440,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -573,7 +576,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 7a7aba33e1..3ed2f79dd0 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1314,6 +1314,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1323,10 +1325,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1391,8 +1395,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 5f49554ea0..68d1c37230 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -109,6 +109,12 @@ typedef struct LogicalDecodingContext
TransactionId write_xid;
/* Are we processing the end LSN of a transaction? */
bool end_xact;
+
+ /*
+ * True if the logical decoding context being used for the creation
+ * of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] master_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (13.0K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/6-master_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From f06b465ed2da959fd1d89dfc17ba5134a00e620a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
contrib/test_decoding/meson.build | 1 +
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 +--
src/backend/replication/logical/snapbuild.c | 29 +++++++++---
src/include/replication/snapbuild.h | 1 +
7 files changed, 122 insertions(+), 10 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index c7ce603706..a4ba1a509a 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -8,7 +8,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
spill slot truncate stream stats twophase twophase_stream
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
- twophase_snapshot slot_creation_error catalog_change_snapshot
+ twophase_snapshot slot_creation_error catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index f1548c0faf..f643dc81a2 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -62,6 +62,7 @@ tests += {
'concurrent_stream',
'twophase_snapshot',
'slot_creation_error',
+ 'skip_snapshot_restore',
],
'regress_args': [
'--temp-config', files('logical.conf'),
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 99f31849bb..f8ef5d56d2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -152,6 +152,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -212,7 +213,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ need_full_snapshot, in_create, slot->data.two_phase_at);
ctx->reorder->private_data = ctx;
@@ -438,7 +439,7 @@ CreateInitDecodingContext(const char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -592,7 +593,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index e37e22f441..ae676145e6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -189,6 +189,14 @@ struct SnapBuild
/* Indicates if we are building full snapshot or just catalog one. */
bool building_full_snapshot;
+ /*
+ * Indicates if we are using the snapshot builder for the creation of a
+ * logical replication slot. If it's true, the start point for decoding
+ * changes is not determined yet. So we skip snapshot restores to properly
+ * find the start point. See SnapBuildFindSnapshot() for details.
+ */
+ bool in_slot_creation;
+
/*
* Snapshot that's valid to see the catalog state seen at this moment.
*/
@@ -317,6 +325,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at)
{
MemoryContext context;
@@ -347,6 +356,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->initial_xmin_horizon = xmin_horizon;
builder->start_decoding_at = start_lsn;
+ builder->in_slot_creation = in_slot_creation;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
@@ -1327,10 +1337,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1395,8 +1407,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !builder->in_slot_creation &&
SnapBuildRestore(builder, lsn))
{
/* there won't be any state to cleanup */
@@ -1580,7 +1597,7 @@ typedef struct SnapBuildOnDisk
offsetof(SnapBuildOnDisk, version)
#define SNAPBUILD_MAGIC 0x51A1E001
-#define SNAPBUILD_VERSION 5
+#define SNAPBUILD_VERSION 6
/*
* Store/Load a snapshot from disk, depending on the snapshot builder's state.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index a3360a1c5e..caa5113ff8 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -62,6 +62,7 @@ extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *reorder,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
bool need_full_snapshot,
+ bool in_slot_creation,
XLogRecPtr two_phase_at);
extern void FreeSnapshotBuilder(SnapBuild *builder);
--
2.39.3
[application/octet-stream] REL13_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/7-REL13_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From d21804242ce36ec69766fd4344b537d39c8e06f6 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 6b9f24e52d..70bd983dc2 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -120,6 +120,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogReaderRoutine *xl_routine,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -198,6 +199,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -327,7 +330,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
xl_routine, prepare_write, do_write,
update_progress);
@@ -429,7 +432,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, xl_routine, prepare_write,
+ fast_forward, false, xl_routine, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 31caad8855..7546de9676 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1307,6 +1307,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1316,10 +1318,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1384,8 +1388,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 14d71fd136..6dcf6bc45a 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -89,6 +89,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the creation
+ * of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
[application/octet-stream] REL12_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch (11.3K, ../../CAD21AoDWk3+geXRz7TJdjcoiACYWedPEdoY6wnAfOZknQP5d=Q@mail.gmail.com/8-REL12_v4-0001-Fix-possibility-of-logical-decoding-partial-trans.patch)
download | inline diff:
From 6d82d2b5b35aa302a0ad31bbccd11ae3f81b4619 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 5 Jul 2024 10:29:09 +0900
Subject: [PATCH v4] Fix possibility of logical decoding partial transaction
changes.
When creating and initializing a logical slot, the restart_lsn is set
to the latest WAL insertion point (or the latest replay point on
standbys). Subsequently, WAL records are decoded from that point to
find the start point for extracting changes in the
DecodingContextFindStartpoint() function. Since the initial
restart_lsn could be in the middle of a transaction, the start point
must be a consistent point where we won't see the data for partial
transactions.
Previously, when not building a full snapshot, serialized snapshots
were restored, and the SnapBuild jumps to the consistent state even
while finding the start point. Consequently, the slot's restart_lsn
and confirmed_flush could be set to the middle of a transaction. This
could lead to various unexpected consequences. Specifically, there
were reports of logical decoding decoding partial transactions, and
assertion failures occurred because only subtransactions were decoded
without decoding their top-level transaction until decoding the commit
record.
To resolve this issue, the changes prevent restoring the serialized
snapshot and jumping to the consistent state while finding the start
point.
On v17 and HEAD, a flag indicating whether snapshot restores should be
skipped has been added to the SnapBuild struct, and SNAPBUILD_VERSION
has been bumpded.
On backbranches, the flag is stored in the LogicalDecodingContext
instead, preserving on-disk compatibility.
Backpatch to all supported versions.
Reported-by: Drew Callahan
Reviewed-by: Amit Kapila, Hayato Kuroda
Discussion: https://postgr.es/m/2444AA15-D21B-4CCE-8052-52C7C2DAFE5C%40amazon.com
Backpatch-through: 12
---
contrib/test_decoding/Makefile | 3 +-
.../expected/skip_snapshot_restore.out | 45 ++++++++++++++++++
.../specs/skip_snapshot_restore.spec | 46 +++++++++++++++++++
src/backend/replication/logical/logical.c | 7 ++-
src/backend/replication/logical/snapbuild.c | 19 ++++++--
src/include/replication/logical.h | 6 +++
6 files changed, 118 insertions(+), 8 deletions(-)
create mode 100644 contrib/test_decoding/expected/skip_snapshot_restore.out
create mode 100644 contrib/test_decoding/specs/skip_snapshot_restore.spec
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 6ec09ab192..735b7e7653 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -7,7 +7,8 @@ REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
spill slot truncate
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
- oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot
+ oldest_xmin snapshot_transfer subxact_without_top catalog_change_snapshot \
+ skip_snapshot_restore
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
ISOLATION_OPTS = --temp-config $(top_srcdir)/contrib/test_decoding/logical.conf
diff --git a/contrib/test_decoding/expected/skip_snapshot_restore.out b/contrib/test_decoding/expected/skip_snapshot_restore.out
new file mode 100644
index 0000000000..c64dbd9c4e
--- /dev/null
+++ b/contrib/test_decoding/expected/skip_snapshot_restore.out
@@ -0,0 +1,45 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s0_init s0_begin s0_insert1 s1_init s2_checkpoint s2_get_changes_slot0 s0_insert2 s0_commit s1_get_changes_slot0 s1_get_changes_slot1
+step s0_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding');
+?column?
+--------
+init
+(1 row)
+
+step s0_begin: BEGIN;
+step s0_insert1: INSERT INTO tbl VALUES (1);
+step s1_init: SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); <waiting ...>
+step s2_checkpoint: CHECKPOINT;
+step s2_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+step s0_insert2: INSERT INTO tbl VALUES (2);
+step s0_commit: COMMIT;
+step s1_init: <... completed>
+?column?
+--------
+init
+(1 row)
+
+step s1_get_changes_slot0: SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+-----------------------------------------
+BEGIN
+table public.tbl: INSERT: val1[integer]:1
+table public.tbl: INSERT: val1[integer]:2
+COMMIT
+(4 rows)
+
+step s1_get_changes_slot1: SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+data
+----
+(0 rows)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/skip_snapshot_restore.spec b/contrib/test_decoding/specs/skip_snapshot_restore.spec
new file mode 100644
index 0000000000..3f1fb6f02c
--- /dev/null
+++ b/contrib/test_decoding/specs/skip_snapshot_restore.spec
@@ -0,0 +1,46 @@
+# Test that a slot creation skips to restore serialized snapshot to reach
+# the consistent state.
+
+setup
+{
+ DROP TABLE IF EXISTS tbl;
+ CREATE TABLE tbl (val1 integer);
+}
+
+teardown
+{
+ DROP TABLE tbl;
+ SELECT 'stop' FROM pg_drop_replication_slot('slot0');
+ SELECT 'stop' FROM pg_drop_replication_slot('slot1');
+}
+
+session "s0"
+setup { SET synchronous_commit = on; }
+step "s0_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot0', 'test_decoding'); }
+step "s0_begin" { BEGIN; }
+step "s0_insert1" { INSERT INTO tbl VALUES (1); }
+step "s0_insert2" { INSERT INTO tbl VALUES (2); }
+step "s0_commit" { COMMIT; }
+
+session "s1"
+setup { SET synchronous_commit = on; }
+step "s1_init" { SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'test_decoding'); }
+step "s1_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+step "s1_get_changes_slot1" { SELECT data FROM pg_logical_slot_get_changes('slot1', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+session "s2"
+setup { SET synchronous_commit = on ;}
+step "s2_checkpoint" { CHECKPOINT; }
+step "s2_get_changes_slot0" { SELECT data FROM pg_logical_slot_get_changes('slot0', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0'); }
+
+
+# While 'slot1' creation by "s1_init" waits for s0-transaction to commit, the
+# RUNNING_XACTS record is written by "s2_checkpoint" and "s2_get_changes_slot1"
+# serializes consistent snapshots to the disk at LSNs where are before
+# s0-transaction's commit. After s0-transaction commits, "s1_init" resumes but
+# must not restore any serialized snapshots and will reach the consistent state
+# when decoding a RUNNING_XACT record generated after s0-transaction's commit.
+# We check if the get_changes on 'slot1' will not return any s0-transaction's
+# changes as its confirmed_flush_lsn will be after the s0-transaction's commit
+# record.
+permutation "s0_init" "s0_begin" "s0_insert1" "s1_init" "s2_checkpoint" "s2_get_changes_slot0" "s0_insert2" "s0_commit" "s1_get_changes_slot0" "s1_get_changes_slot1"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8b79d67001..e6423b955b 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -123,6 +123,7 @@ StartupDecodingContext(List *output_plugin_options,
TransactionId xmin_horizon,
bool need_full_snapshot,
bool fast_forward,
+ bool in_create,
XLogPageReadCB read_page,
LogicalOutputPluginWriterPrepareWrite prepare_write,
LogicalOutputPluginWriterWrite do_write,
@@ -201,6 +202,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->fast_forward = fast_forward;
+ ctx->in_create = in_create;
+
MemoryContextSwitchTo(old_context);
return ctx;
@@ -329,7 +332,7 @@ CreateInitDecodingContext(char *plugin,
ReplicationSlotSave();
ctx = StartupDecodingContext(NIL, restart_lsn, xmin_horizon,
- need_full_snapshot, false,
+ need_full_snapshot, false, true,
read_page, prepare_write, do_write,
update_progress);
@@ -428,7 +431,7 @@ CreateDecodingContext(XLogRecPtr start_lsn,
ctx = StartupDecodingContext(output_plugin_options,
start_lsn, InvalidTransactionId, false,
- fast_forward, read_page, prepare_write,
+ fast_forward, false, read_page, prepare_write,
do_write, update_progress);
/* call output plugin initialization callback */
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index bdfd1f0228..517fb4d6c0 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1312,6 +1312,8 @@ SnapBuildProcessRunningXacts(SnapBuild *builder, XLogRecPtr lsn, xl_running_xact
static bool
SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *running)
{
+ LogicalDecodingContext *ctx = (LogicalDecodingContext *) builder->reorder->private_data;
+
/* ---
* Build catalog decoding snapshot incrementally using information about
* the currently running transactions. There are several ways to do that:
@@ -1321,10 +1323,12 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
* state while waiting on c)'s sub-states.
*
* b) This (in a previous run) or another decoding slot serialized a
- * snapshot to disk that we can use. Can't use this method for the
- * initial snapshot when slot is being created and needs full snapshot
- * for export or direct use, as that snapshot will only contain catalog
- * modifying transactions.
+ * snapshot to disk that we can use. Can't use this method while finding
+ * the start point for decoding changes as the restart LSN would be an
+ * arbitrary LSN but we need to find the start point to extract changes
+ * where we won't see the data for partial transactions. Also, we cannot
+ * use this method when a slot needs a full snapshot for export or direct
+ * use, as that snapshot will only contain catalog modifying transactions.
*
* c) First incrementally build a snapshot for catalog tuples
* (BUILDING_SNAPSHOT), that requires all, already in-progress,
@@ -1389,8 +1393,13 @@ SnapBuildFindSnapshot(SnapBuild *builder, XLogRecPtr lsn, xl_running_xacts *runn
return false;
}
- /* b) valid on disk state and not building full snapshot */
+
+ /*
+ * b) valid on disk state and while neither building full snapshot nor
+ * creating a slot.
+ */
else if (!builder->building_full_snapshot &&
+ !ctx->in_create &&
SnapBuildRestore(builder, lsn))
{
int nxacts = running->subxcnt + running->xcnt;
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 718080d54a..89f9893b47 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -90,6 +90,12 @@ typedef struct LogicalDecodingContext
bool prepared_write;
XLogRecPtr write_location;
TransactionId write_xid;
+
+ /*
+ * True if the logical decoding context being used for the creation
+ * of a logical replication slot.
+ */
+ bool in_create;
} LogicalDecodingContext;
--
2.39.3
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-12 02:52 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 40+ messages in thread
From: Masahiko Sawada @ 2024-07-12 02:52 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jul 10, 2024 at 2:30 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jul 10, 2024 at 12:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Jul 10, 2024 at 7:52 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > Overall I think the patches are in good shape, so I'm going to push
> > > them tomorrow, barring any objections and further comments.
> > >
> >
> > Agreed. Below are a few minor comments that you might want to consider:
> >
> > 1.
> > @@ -76,6 +77,7 @@ extern SnapBuildState
> > SnapBuildCurrentState(SnapBuild *builder);
> > extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder);
> >
> > extern bool SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
> > +extern void SnapBuildSetFindStartPoint(SnapBuild *builder, bool
> > find_start_point);
> >
> > This is not required in the latest version.
> >
> > 2.
> > + /*
> > + * Indicates if we are using the snapshot builder for the initial creation
> > + * of a logical replication slot.
> >
> > The word 'initial' in the above comment is not required. If you
> > consider this then a similar wording change is required in lower
> > branches as well.
> >
> > 3.
> > HEAD and v17
> > ---------------------
> > - /* b) valid on disk state and not building full snapshot */
> > +
> > + /*
> > + * b) valid on disk state and while neither building full snapshot nor
> > + * finding the start point.
> > + */
> > else if (!builder->building_full_snapshot &&
> > + !builder->in_slot_creation &&
> >
> > V16 and below
> > ---------------------
> > - /* b) valid on disk state and not building full snapshot */
> > +
> > + /*
> > + * b) valid on disk state and neither building full snapshot nor while
> > + * creating a slot.
> > + */
> > else if (!builder->building_full_snapshot &&
> > + !ctx->in_create &&
> >
> > Isn't it better to use the same comment in both places?
>
> Thank you for reviewing the patches! I agreed with all the points.
> I've attached the updated patches.
>
Pushed, bb19b70081 (and all supported branches).
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: Potential data loss due to race condition during logical replication slot creation
@ 2024-07-12 03:42 Michael Paquier <michael@paquier.xyz>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 40+ messages in thread
From: Michael Paquier @ 2024-07-12 03:42 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Callahan, Drew <callaan@amazon.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, Jul 12, 2024 at 11:52:20AM +0900, Masahiko Sawada wrote:
> Pushed, bb19b70081 (and all supported branches).
Thanks, Sawada-san.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZpCmD0B_s8B30IjW@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 40+ messages in thread
end of thread, other threads:[~2024-07-12 03:42 UTC | newest]
Thread overview: 40+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-02-01 22:28 Potential data loss due to race condition during logical replication slot creation Callahan, Drew <callaan@amazon.com>
2024-02-02 00:53 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-02-03 13:48 ` vignesh C <vignesh21@gmail.com>
2024-02-05 02:12 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-02-03 03:12 Re: Potential data loss due to race condition during logical replication slot creation Callahan, Drew <callaan@amazon.com>
2024-02-05 02:06 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-02-08 05:54 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-02-19 09:12 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-13 09:34 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-13 09:47 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-18 09:08 ` Amit Kapila <amit.kapila16@gmail.com>
2024-03-19 02:16 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-19 02:33 ` Amit Kapila <amit.kapila16@gmail.com>
2024-03-19 04:39 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-19 16:42 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-27 07:54 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-27 08:26 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-27 10:37 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-03-18 09:18 ` Amit Kapila <amit.kapila16@gmail.com>
2024-04-16 03:56 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-06-20 06:19 ` Amit Kapila <amit.kapila16@gmail.com>
2024-06-21 06:46 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-06-24 03:54 ` Amit Kapila <amit.kapila16@gmail.com>
2024-06-24 05:01 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-06-25 04:24 ` Amit Kapila <amit.kapila16@gmail.com>
2024-06-25 08:26 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-06-26 09:17 ` Amit Kapila <amit.kapila16@gmail.com>
2024-07-05 14:52 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-07 23:56 ` Michael Paquier <michael@paquier.xyz>
2024-07-08 03:01 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-08 09:01 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2024-07-08 14:02 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-09 05:29 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-09 09:27 ` Amit Kapila <amit.kapila16@gmail.com>
2024-07-10 01:15 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-10 02:21 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-10 03:43 ` Amit Kapila <amit.kapila16@gmail.com>
2024-07-10 05:30 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-12 02:52 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-07-12 03:42 ` Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox