agora inbox for pgsql-committers@postgresql.org
help / color / mirror / Atom feedpgsql: Add CONCURRENTLY option to REPACK
3+ messages / 2 participants
[nested] [flat]
* pgsql: Add CONCURRENTLY option to REPACK
@ 2026-04-06 19:56 Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 1 reply; 3+ messages in thread
From: Álvaro Herrera @ 2026-04-06 19:56 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Add CONCURRENTLY option to REPACK
When this flag is specified, REPACK no longer acquires access-exclusive
lock while the new copy of the table is being created; instead, it
creates the initial copy under share-update-exclusive lock only (same as
vacuum, etc), and it follows an MVCC snapshot; it sets up a replication
slot starting at that snapshot, and uses a concurrent background worker
to do logical decoding starting at the snapshot to populate a stash of
concurrent data changes. Those changes can then be re-applied to the
new copy of the table just before swapping the relfilenodes.
Applications can continue to access the original copy of the table
normally until just before the swap, which is the only point at which
the access-exclusive lock is needed.
There are some loose ends in this commit:
1. concurrent repack needs its own replication slot in order to apply
logical decoding, which are a scarce resource and easy to run out of.
2. due to the way the historic snapshot is initially set up, only one
REPACK process can be running at any one time on the whole system.
3. there's a danger of deadlocking (and thus abort) due to the lock
upgrade required at the final phase.
These issues will be addressed in upcoming commits.
The design and most of the code are by Antonin Houska, heavily based on
his own pg_squeeze third-party implementation.
Author: Antonin Houska <ah@cybertec.at>
Co-authored-by: Mihail Nikalayeu <mihailnikalayeu@gmail.com>
Co-authored-by: Álvaro Herrera <alvherre@kurilemu.de>
Reviewed-by: Matthias van de Meent <boekewurm+postgres@gmail.com>
Reviewed-by: Srinath Reddy Sadipiralla <srinath2133@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Jim Jones <jim.jones@uni-muenster.de>
Reviewed-by: Robert Treat <rob@xzilla.net>
Reviewed-by: Noriyoshi Shinoda <noriyoshi.shinoda@hpe.com>
Reviewed-by: vignesh C <vignesh21@gmail.com>
Discussion: https://postgr.es/m/5186.1706694913@antos
Discussion: https://postgr.es/m/202507262156.sb455angijk6@alvherre.pgsql
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/28d534e2ae0ac888b5460f977a10cd9bb017ef98
Modified Files
--------------
doc/src/sgml/monitoring.sgml | 37 +-
doc/src/sgml/mvcc.sgml | 12 +-
doc/src/sgml/ref/repack.sgml | 118 +-
src/Makefile | 1 +
src/backend/access/heap/heapam.c | 26 +-
src/backend/access/heap/heapam_handler.c | 290 +++-
src/backend/access/heap/rewriteheap.c | 6 +-
src/backend/catalog/system_views.sql | 19 +-
src/backend/commands/Makefile | 1 +
src/backend/commands/matview.c | 1 +
src/backend/commands/meson.build | 1 +
src/backend/commands/repack.c | 1835 ++++++++++++++++++--
src/backend/commands/repack_worker.c | 533 ++++++
src/backend/commands/tablecmds.c | 1 +
src/backend/commands/vacuum.c | 12 +-
src/backend/libpq/pqmq.c | 5 +
src/backend/meson.build | 1 +
src/backend/postmaster/bgworker.c | 5 +
src/backend/replication/logical/decode.c | 28 +-
src/backend/replication/pgrepack/Makefile | 32 +
src/backend/replication/pgrepack/meson.build | 18 +
src/backend/replication/pgrepack/pgrepack.c | 287 +++
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/activity/wait_event_names.txt | 1 +
src/bin/psql/tab-complete.in.c | 4 +-
src/include/access/heapam_xlog.h | 2 +
src/include/access/tableam.h | 15 +-
src/include/commands/progress.h | 17 +-
src/include/commands/repack.h | 15 +-
src/include/commands/repack_internal.h | 125 ++
src/include/replication/decode.h | 4 +
src/include/storage/lockdefs.h | 4 +-
src/include/storage/procsignal.h | 1 +
src/makefiles/Makefile.cygwin | 2 +
src/makefiles/Makefile.win32 | 2 +
src/test/modules/injection_points/Makefile | 2 +
.../modules/injection_points/expected/repack.out | 113 ++
.../injection_points/expected/repack_toast.out | 65 +
src/test/modules/injection_points/meson.build | 2 +
.../modules/injection_points/specs/repack.spec | 142 ++
.../injection_points/specs/repack_toast.spec | 112 ++
src/test/regress/expected/cluster.out | 8 +
src/test/regress/expected/rules.out | 19 +-
src/test/regress/sql/cluster.sql | 6 +
src/tools/pgindent/typedefs.list | 6 +
46 files changed, 3682 insertions(+), 260 deletions(-)
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: pgsql: Add CONCURRENTLY option to REPACK
@ 2026-04-07 15:24 Aleksander Alekseev <aleksander@tigerdata.com>
parent: Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 1 reply; 3+ messages in thread
From: Aleksander Alekseev @ 2026-04-07 15:24 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org; +Cc: Álvaro Herrera <alvherre@kurilemu.de>
Hi Alvaro,
> Add CONCURRENTLY option to REPACK
>
> [...]
I noticed that restore_tuple() checks for an insufficient number of
attributes stored separately, but doesn't check for an unlikely event
when there are too many attributes. I don't think this is intentional.
Any chance we want to fix this?
--
Best regards,
Aleksander Alekseev
Attachments:
[text/x-patch] v1-0001-Use-stricter-checks-in-restore_tuple.patch (865B, ../../CAJ7c6TMSF7cANU8nEJ9E28EvU74tE4H7AzT292Rt3ZuHqqxq8w@mail.gmail.com/2-v1-0001-Use-stricter-checks-in-restore_tuple.patch)
download | inline diff:
From 7c0151e1bc272b67306801b287d1546309f4f422 Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <aleksander@tigerdata.com>
Date: Tue, 7 Apr 2026 18:13:30 +0300
Subject: [PATCH v1] Use stricter checks in restore_tuple()
---
src/backend/commands/repack.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 20dad22c4b7..6919c6c8536 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -2741,6 +2741,11 @@ restore_tuple(BufFile *file, Relation relation, TupleTableSlot *slot)
errcode(ERRCODE_DATA_CORRUPTED),
errmsg("insufficient number of attributes stored separately"));
}
+
+ if (natt_ext != 0)
+ ereport(ERROR,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("unexpected number of attributes stored separately"));
}
}
--
2.43.0
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: pgsql: Add CONCURRENTLY option to REPACK
@ 2026-06-16 12:30 Álvaro Herrera <alvherre@kurilemu.de>
parent: Aleksander Alekseev <aleksander@tigerdata.com>
0 siblings, 0 replies; 3+ messages in thread
From: Álvaro Herrera @ 2026-06-16 12:30 UTC (permalink / raw)
To: Aleksander Alekseev <aleksander@tigerdata.com>; +Cc: pgsql-committers@lists.postgresql.org
Hi,
On 2026-Apr-07, Aleksander Alekseev wrote:
> I noticed that restore_tuple() checks for an insufficient number of
> attributes stored separately, but doesn't check for an unlikely event
> when there are too many attributes. I don't think this is intentional.
> Any chance we want to fix this?
Yeah, that's a reasonable idea. I have added the check, thanks.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"No me acuerdo, pero no es cierto. No es cierto, y si fuera cierto,
no me acuerdo." (Augusto Pinochet a una corte de justicia)
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2026-06-16 12:30 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-04-06 19:56 pgsql: Add CONCURRENTLY option to REPACK Álvaro Herrera <alvherre@kurilemu.de>
2026-04-07 15:24 ` Aleksander Alekseev <aleksander@tigerdata.com>
2026-06-16 12:30 ` Álvaro Herrera <alvherre@kurilemu.de>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox