public inbox for [email protected]
help / color / mirror / Atom feedFrom: Zhijie Hou (Fujitsu) <[email protected]>
To: Alvaro Herrera <[email protected]>
To: Amit Kapila <[email protected]>
Cc: Hayato Kuroda (Fujitsu) <[email protected]>
Cc: Antonin Houska <[email protected]>
Cc: Srinath Reddy Sadipiralla <[email protected]>
Cc: Mihail Nikalayeu <[email protected]>
Cc: Matthias van de Meent <[email protected]>
Cc: Pg Hackers <[email protected]>
Cc: Robert Treat <[email protected]>
Subject: RE: Adding REPACK [concurrently]
Date: Fri, 10 Apr 2026 10:53:53 +0000
Message-ID: <TYRPR01MB14195633567DA00ABD42570B794592@TYRPR01MB14195.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <[email protected]>
References: <CAA4eK1Jwguq90CC8nxOqan+raCS8WisB=Bmb1AmK8rcvtj8GPg@mail.gmail.com>
<[email protected]>
Hi,
When testing REPACK concurrently, I noticed that all WALs are retained from
the moment REPACK begins copying data to the new table until the command
finishes replaying concurrent changes on the new table and stops the repack
decoding worker.
I understand the reason: the REPACK command itself starts a long-running
transaction, and logical decoding does not advance restart_lsn beyond the
oldest running transaction's start position. As a result, slot.restart_lsn
remains unchanged, preventing the checkpointer from recycling WALs.
However, since REPACK can run for a long time (hours or even days), I'd like
to confirm whether this is expected behavior or if we plan to improve it
in the future ? And additionally, IIUC, REPACK without using concurrent option
does not have this issue.
Given that we do not restart a REPACK, I think the repack decoding worker
should be able to advance restart_lsn each time after writing changes
(similar to how a physical slot behaves). To illustrate this, I've written
a patch (attached) that implements this approach, and it works fine for me.
BTW, catalog_xmin also won't advance, but that seems not a big issue as
the REPACK transaction itself also holds a snapshot that retains catalog tuples,
so advancing catalog_xmin wouldn't change the situation anyway.
Thoughts ?
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v1-0001-Allow-old-WALs-to-be-removed-during-REPACK-CONCUR.patch (2.3K, 2-v1-0001-Allow-old-WALs-to-be-removed-during-REPACK-CONCUR.patch)
download | inline diff:
From 3b4c9a890fdeb854949a36b6be8f1fc00a5edafe Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Fri, 10 Apr 2026 16:24:55 +0800
Subject: [PATCH v1] Allow old WALs to be removed during REPACK CONCURRENTLY
---
src/backend/commands/repack_worker.c | 14 +++++++++++++-
src/backend/replication/logical/logical.c | 4 +++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
index 5bd020e0184..50f000ca6df 100644
--- a/src/backend/commands/repack_worker.c
+++ b/src/backend/commands/repack_worker.c
@@ -394,12 +394,24 @@ decode_concurrent_changes(LogicalDecodingContext *ctx,
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance.
+ * system that the slot can advance.
+ *
+ * Once REPACK begins copying data to the new table, the logical
+ * decoding machinery prevents the slot from advancing beyond the
+ * oldest running transaction (which is the REPACK transaction
+ * itself). As a result, restart_lsn and catalog_xmin can no longer
+ * advance automatically.
+ *
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed. We do not
+ * advance catalog_xmin here because the REPACK transaction anyway
+ * holds a snapshot that prevents catalog tuple removal.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
if (segno_new != repack_current_segment)
{
+ LogicalIncreaseRestartDecodingForSlot(end_lsn, end_lsn);
LogicalConfirmReceivedLocation(end_lsn);
elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
(uint32) (end_lsn >> 32), (uint32) end_lsn);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d8e02c53558..baa639835f7 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1913,8 +1913,10 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
}
else
{
--
2.53.0.windows.2
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: RE: Adding REPACK [concurrently]
In-Reply-To: <TYRPR01MB14195633567DA00ABD42570B794592@TYRPR01MB14195.jpnprd01.prod.outlook.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox