public inbox for [email protected]
help / color / mirror / Atom feedRe: Adding REPACK [concurrently]
156+ messages / 19 participants
[nested] [flat]
* Re: Adding REPACK [concurrently]
@ 2026-04-01 14:55 Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-04-01 14:55 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello,
i was fuzz testing v48 , and found a crash when REPACK (concurrently)
itself errors out,
1) while running
create table test(id int);
REPACK (concurrently) test;
TBH i didn't knew this, sometimes it's better to not know "rules" ;)
[NOTE: maybe we should add that we can't run
REPACK (concurrently) on table without identity index or primary key into
repack.sgml]
ERROR: cannot process relation "test"
2026-04-01 19:06:31.211 IST [2495575] HINT: Relation "test" has no
identity index.
2026-04-01 19:06:31.211 IST [2495575] STATEMENT: repack (concurrently)
test;
TRAP: failed Assert("proc->statusFlags ==
ProcGlobal->statusFlags[proc->pgxactoff]"), File: "procarray.c", Line: 719,
PID: 2495575
postgres: srinath postgres [local]
REPACK(ExceptionalCondition+0x98)[0xaaaaad938d84]
postgres: srinath postgres [local]
REPACK(ProcArrayEndTransaction+0x1f0)[0xaaaaad6c15fc]
postgres: srinath postgres [local] REPACK(+0x210cf0)[0xaaaaad190cf0]
postgres: srinath postgres [local] REPACK(+0x2117e4)[0xaaaaad1917e4]
postgres: srinath postgres [local]
REPACK(AbortCurrentTransaction+0x10)[0xaaaaad191740]
postgres: srinath postgres [local]
REPACK(PostgresMain+0x568)[0xaaaaad7116e4]
postgres: srinath postgres [local] REPACK(+0x786ae0)[0xaaaaad706ae0]
postgres: srinath postgres [local]
REPACK(postmaster_child_launch+0x1f0)[0xaaaaad5d719c]
postgres: srinath postgres [local] REPACK(+0x65ea98)[0xaaaaad5dea98]
postgres: srinath postgres [local] REPACK(+0x65b650)[0xaaaaad5db650]
postgres: srinath postgres [local]
REPACK(PostmasterMain+0x1564)[0xaaaaad5dae1c]
postgres: srinath postgres [local] REPACK(main+0x3dc)[0xaaaaad466348]
/lib/aarch64-linux-gnu/libc.so.6(+0x284c4)[0xffffb40d84c4]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffffb40d8598]
postgres: srinath postgres [local] REPACK(_start+0x30)[0xaaaaad06ddf0]
2026-04-01 19:06:31.800 IST [2494560] LOG: client backend (PID 2495575)
was terminated by signal 6: Aborted
2) And when running REPACK (concurrently) on the same table while already a
repack was running
on the same table ,just to verify the deadlock occurs and gets errored out
that
"could not wait for concurrent REPACK" but instead got the same crash.
ERROR: could not wait for concurrent REPACK
2026-04-01 12:55:39.481 IST [2397660] DETAIL: Process 2397660 waits for
REPACK running on process 2397307
2026-04-01 12:55:39.481 IST [2397660] CONTEXT: waiting for
ShareUpdateExclusiveLock on relation 16385 of database 5
2026-04-01 12:55:39.481 IST [2397660] STATEMENT: repack (concurrently)
stress_victim ;
2026-04-01 12:55:39.497 IST [2397151] LOG: checkpoint complete: time:
wrote 2056 buffers (12.5%), wrote 0 SLRU buffers; 0 WAL file(s) added, 0
removed, 0 recycled; write=206.804 s, sync=0.003 s, total=861.616 s; sync
files=17, longest=0.002 s, average=0.001 s; distance=318978 kB,
estimate=515341 kB; lsn=2/02810A60, redo lsn=2/02810910
TRAP: failed Assert("proc->statusFlags ==
ProcGlobal->statusFlags[proc->pgxactoff]"), File: "procarray.c", Line: 719,
PID: 2397660
postgres: srinath postgres [local]
REPACK(ExceptionalCondition+0x98)[0xaaaae7d58d84]
postgres: srinath postgres [local]
REPACK(ProcArrayEndTransaction+0x1f0)[0xaaaae7ae15fc]
postgres: srinath postgres [local] REPACK(+0x210cf0)[0xaaaae75b0cf0]
postgres: srinath postgres [local] REPACK(+0x2117e4)[0xaaaae75b17e4]
postgres: srinath postgres [local]
REPACK(AbortCurrentTransaction+0x10)[0xaaaae75b1740]
postgres: srinath postgres [local]
REPACK(PostgresMain+0x568)[0xaaaae7b316e4]
postgres: srinath postgres [local] REPACK(+0x786ae0)[0xaaaae7b26ae0]
postgres: srinath postgres [local]
REPACK(postmaster_child_launch+0x1f0)[0xaaaae79f719c]
postgres: srinath postgres [local] REPACK(+0x65ea98)[0xaaaae79fea98]
postgres: srinath postgres [local] REPACK(+0x65b650)[0xaaaae79fb650]
postgres: srinath postgres [local]
REPACK(PostmasterMain+0x1564)[0xaaaae79fae1c]
postgres: srinath postgres [local] REPACK(main+0x3dc)[0xaaaae7886348]
/lib/aarch64-linux-gnu/libc.so.6(+0x284c4)[0xffff9ec984c4]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffff9ec98598]
postgres: srinath postgres [local] REPACK(_start+0x30)[0xaaaae748ddf0]
2026-04-01 12:58:18.198 IST [2397147] LOG: client backend (PID 2397660)
was terminated by signal 6: Aborted
the reason for this crash was ProcGlobal->statusFlags was not initialized
during the start of ExecRepack , earlier Abort before reaching the proper
initialization of ProcGlobal->statusFlags which was done in rebuild_relation
caused this assert failure in ProcArrayEndTransaction.
Here's the diff to solve this crash.
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 29ba49744eb..d44092a407a 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -284,7 +284,23 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool
isTopLevel)
* that others can conflict with.
*/
if ((params.options & CLUOPT_CONCURRENT) != 0)
+ {
+ /*
+ * Do not let other backends wait for our completion during their
+ * setup of logical replication. Unlike logical replication publisher,
+ * we will have XID assigned, so the other backends - whether
+ * walsenders involved in logical replication or regular backends
+ * executing also REPACK (CONCURRENTLY) - would have to wait for our
+ * completion before they can build their initial snapshot. It is o.k.
+ * for any decoding backend to ignore us because we do not change
+ * tuple descriptor of any table, and the data changes we write should
+ * not be decoded by other backends.
+ */
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
MyProc->statusFlags |= PROC_IN_CONCURRENT_REPACK;
+ ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags;
+ LWLockRelease(ProcArrayLock);
+ }
/*
* If a single relation is specified, process it and we're done ... unless
@@ -988,22 +1004,6 @@ rebuild_relation(Relation OldHeap, Relation index,
bool verbose,
if (concurrent)
{
- /*
- * Do not let other backends wait for our completion during their
- * setup of logical replication. Unlike logical replication publisher,
- * we will have XID assigned, so the other backends - whether
- * walsenders involved in logical replication or regular backends
- * executing also REPACK (CONCURRENTLY) - would have to wait for our
- * completion before they can build their initial snapshot. It is o.k.
- * for any decoding backend to ignore us because we do not change
- * tuple descriptor of any table, and the data changes we write should
- * not be decoded by other backends.
- */
- LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- MyProc->statusFlags |= PROC_IN_CONCURRENT_REPACK;
- ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags;
- LWLockRelease(ProcArrayLock);
-
/*
* The worker needs to be member of the locking group we're the leader
* of. We ought to become the leader before the worker starts. The
Thoughts?
--
Thanks,
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
@ 2026-04-01 17:06 ` Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-02 00:19 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-01 17:06 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Srinath Reddy Sadipiralla <[email protected]> wrote:
> i was fuzz testing v48 , and found a crash when REPACK (concurrently) itself errors out,
> 1) while running
>
> create table test(id int);
> REPACK (concurrently) test;
>
> TBH i didn't knew this, sometimes it's better to not know "rules" ;)
> [NOTE: maybe we should add that we can't run
> REPACK (concurrently) on table without identity index or primary key into repack.sgml]
>
> ERROR: cannot process relation "test"
> 2026-04-01 19:06:31.211 IST [2495575] HINT: Relation "test" has no identity index.
> 2026-04-01 19:06:31.211 IST [2495575] STATEMENT: repack (concurrently) test;
> TRAP: failed Assert("proc->statusFlags == ProcGlobal->statusFlags[proc->pgxactoff]"), File: "procarray.c", Line: 719, PID: 2495575
> Here's the diff to solve this crash.
Thanks. Attached here is v48-0006 fixed.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-01 21:52 ` Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-01 21:52 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-01, Antonin Houska wrote:
> Thanks. Attached here is v48-0006 fixed.
Ah thanks, I incorporated this in the series and rebased, and here's
also a quick and simple additional patch that adds a GUC
max_repack_replication_slots which are especially there to support
REPACK.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-03 12:08 ` Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-03 12:08 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-01, Antonin Houska wrote:
>
> > Thanks. Attached here is v48-0006 fixed.
>
> Ah thanks, I incorporated this in the series and rebased, and here's
> also a quick and simple additional patch that adds a GUC
> max_repack_replication_slots which are especially there to support
> REPACK.
This is an alternative implementation of 0006, allowing one backend running
REPACK (CONCURRENTLY) in a database, instead of one backend in a cluster.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-03 14:59 ` Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-03 14:59 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-03, Antonin Houska wrote:
> This is an alternative implementation of 0006, allowing one backend running
> REPACK (CONCURRENTLY) in a database, instead of one backend in a cluster.
Thanks! so I'm removing the previous one and taking this one. Here's a
v50:
- In testing, I noticed that we could sometimes request a Flush for a
WAL position that hasn't been written yet. This is due to my
replacing the original code that wrote a dummy xlog record that we
could flush, with a call to XLogGetInsertRecPtr(). So we'd get an
error like
LOG: request to flush past end of generated WAL; request 0/15CF0018, current position 0/15CF000
Antonin promptly noticed that this is because XLogGetInsertRecPtr()
gets the LSN past the segment header, which is 18 bytes wrong. So the
fix here is to use XLogGetInsertEndRecPtr() instead.
- My testing also uncovered a problem with exclusion constraints; tables
with them would fail to repack like
ERROR: exclusion constraint record missing for rel temporal_fk_mltrng2mltrng_pk_repacknew
Antonin sent a patch to create copies of the constraints on the
transient index, which seems like it fixes the problem. Those
constraints are obviously discarded together with the transient index.
- I polished the patch to reserve replication slots for REPACK. Given
the new implementation of 0006 that was submitted implies that we can
now run multiple repacks concurrently, I changed the default of 1 to 5.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Here's a general engineering tip: if the non-fun part is too complex for you
to figure out, that might indicate the fun part is too ambitious." (John Naylor)
https://postgr.es/m/CAFBsxsG4OWHBbSDM%3DsSeXrQGOtkPiOEOuME4yD7Ce41NtaAD9g%40mail.gmail.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-03 19:31 ` Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 13:16 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-03 19:31 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-03, Alvaro Herrera wrote:
> - I polished the patch to reserve replication slots for REPACK. Given
> the new implementation of 0006 that was submitted implies that we can
> now run multiple repacks concurrently, I changed the default of 1 to 5.
Srinath let me know that this new part was causing CI failures on
Windows. This version v51 should be okay (or, at least, it passes for
me on CI). Additional changes worth mentioning:
- I think it's nicer for the index_create() API to get a bit to indicate
suppression of progress reporting; so existing callers don't need to
do anything. I guess this is mostly a matter of taste.
- I incorporated Srinath's fix for the PreventInTransactionBlock block.
- When CheckSlotRequirements() is to complain about
"max_replication_slots or max_repack_replication_slots", it seems
actually nicer to say exactly which one of these is the cause of the
problem. This is easy to change; patch 0010 does it; it requires
passing down a "repack" flag all the way from
CheckLogicalDecodingRequirements() and it needs to add an argument to
CreateInitDecodingContext(), which is perhaps not so great. On the
whole I'm inclined to do it anyway, but I'm about equally happy to
leave it alone. This is what would change:
original:
ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("replication slots can only be used if \"%s\" > 0 or \"%s\" > 0",
- "max_replication_slots", "max_repack_replication_slots")));
patched:
ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replication slots can only be used if \"%s\" > 0",
+ repack ? "max_repack_replication_slots" : "max_replication_slots"));
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-03 19:37 ` Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-03 19:37 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
Sorry -- there's a doc build failure also in CI. Fixed here.
Regards
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Use it up, wear it out, make it do, or do without"
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-04 06:20 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-04-04 06:20 UTC (permalink / raw)
To: 'Alvaro Herrera' <[email protected]>; Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Dear Álvaro,
While testing REPACK CONCURRENTLY command with xid_wraparound module, noticed that
wraparound-autovac worker was terminated with an error like below.
``
ERROR: could not wait for concurrent REPACK
DETAIL: Process 41512 waits for REPACK running on process 41027
CONTEXT: waiting for ShareUpdateExclusiveLock on relation 16384 of database 5
automatic vacuum of table "postgres.public.test"
```
The behavior is different from other commands, like LOCK and maybe normal REPACK.
In these cases the autovac worker would wait till the command fails.
Is it an intentional behavior? If so, what is the advantage that we terminate the
failsafe vacuum?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
@ 2026-04-04 10:19 ` 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: 'Alvaro Herrera' @ 2026-04-04 10:19 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello,
On 2026-Apr-04, Hayato Kuroda (Fujitsu) wrote:
> While testing REPACK CONCURRENTLY command with xid_wraparound module, noticed that
> wraparound-autovac worker was terminated with an error like below.
>
> ``
> ERROR: could not wait for concurrent REPACK
> DETAIL: Process 41512 waits for REPACK running on process 41027
> CONTEXT: waiting for ShareUpdateExclusiveLock on relation 16384 of database 5
> automatic vacuum of table "postgres.public.test"
> ```
>
> The behavior is different from other commands, like LOCK and maybe normal REPACK.
> In these cases the autovac worker would wait till the command fails.
>
> Is it an intentional behavior? If so, what is the advantage that we terminate the
> failsafe vacuum?
Hmm, this is intentional; see here:
https://postgr.es/m/[email protected]
Note that in order for this to happen, the autovacuum must be starting
when the repack is already underway. The theory behind this behavior is
that the autovacuum run is not useful anyway: its purpose is to advance
the freeze xmin/multixact, but the repack is also going to do that.
Once repack is done, autovacuum can assess again whether an emergency
vacuum is needed, and launch a worker in that case.
Do you think it would be better if we allowed autovacuum to continue?
I'm not 100% myself of this new behavior, and it would be very good to
give it some more thought.
I suppose it's unfortunate that autovacuum launcher is going to try
again and again to get workers to process that table, and they are going
to be killed over and over. Maybe it would be better to have autovac
ignore those tables.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Para tener más hay que desear menos"
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
@ 2026-04-06 05:24 ` Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-04-06 05:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Sat, Apr 4, 2026 at 3:49 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2026-Apr-04, Hayato Kuroda (Fujitsu) wrote:
>
> > While testing REPACK CONCURRENTLY command with xid_wraparound module, noticed that
> > wraparound-autovac worker was terminated with an error like below.
> >
> > ``
> > ERROR: could not wait for concurrent REPACK
> > DETAIL: Process 41512 waits for REPACK running on process 41027
> > CONTEXT: waiting for ShareUpdateExclusiveLock on relation 16384 of database 5
> > automatic vacuum of table "postgres.public.test"
> > ```
> >
> > The behavior is different from other commands, like LOCK and maybe normal REPACK.
> > In these cases the autovac worker would wait till the command fails.
> >
> > Is it an intentional behavior? If so, what is the advantage that we terminate the
> > failsafe vacuum?
>
> Hmm, this is intentional; see here:
> https://postgr.es/m/[email protected]
> Note that in order for this to happen, the autovacuum must be starting
> when the repack is already underway. The theory behind this behavior is
> that the autovacuum run is not useful anyway: its purpose is to advance
> the freeze xmin/multixact, but the repack is also going to do that.
> Once repack is done, autovacuum can assess again whether an emergency
> vacuum is needed, and launch a worker in that case.
>
But won't it delay in update of datfrozenxid/datminmxid? For example,
say repack errored out due to some reason, won't it further delay the
update to datfrozenxid/datminmxid which in turn can delay truncate of
clog. Is it possible that after repack errored out the launcher delays
in picking the same table again which further add to such a delay?
> Do you think it would be better if we allowed autovacuum to continue?
>
IIUC, the disadvantage of letting it continue is that if repack is
successful then we have unnecessarily occupied the worker which won't
do anything useful. If we want to not let autovacuum continue then we
should somehow deal with boundary cases which shouldn't lead to delay
in making progress to update frozen xids.
> I'm not 100% myself of this new behavior, and it would be very good to
> give it some more thought.
>
>
> I suppose it's unfortunate that autovacuum launcher is going to try
> again and again to get workers to process that table, and they are going
> to be killed over and over. Maybe it would be better to have autovac
> ignore those tables.
>
I feel that would be better at least when we know that the repack
concurrently command is already in progress. It can help avoid
launching workers again and again, especially when repack concurrently
command is going to take a long time.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-07 21:38 ` Alvaro Herrera <[email protected]>
2026-04-10 07:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 21:38 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-06, Amit Kapila wrote:
> On Sat, Apr 4, 2026 at 3:49 PM Alvaro Herrera <[email protected]> wrote:
> > I suppose it's unfortunate that autovacuum launcher is going to try
> > again and again to get workers to process that table, and they are going
> > to be killed over and over. Maybe it would be better to have autovac
> > ignore those tables.
>
> I feel that would be better at least when we know that the repack
> concurrently command is already in progress. It can help avoid
> launching workers again and again, especially when repack concurrently
> command is going to take a long time.
Okay. I implemented that now, and here it is. 0001 is as before; 0002
creates shared memory for repack and has autovacuum recheck each table
there before processing it.
Having implemented it, I'm not sure the resulting behavior is all that
different from before. I mean, the only difference is that the worker
is going to see the table listed in repack shmem and skip it; as opposed
to trying to lock it and be terminated by the deadlock detector one
second later, at which point it continues with the next table on its
list. So it's some wasted work to set up the autovac work, but nothing
more.
However, there's also the point Andres just made in [1] which basically
kills this idea. So I'm withdrawing this proposal. Still, having
written it, I thought it'd be better to get it archived.
[1] https://postgr.es/m/4n4q3preb3lgyhpzstebhux7b2aojhsw7gik4ivaznyggiezrs@lrznutssxlh2
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
“Cuando no hay humildad las personas se degradan” (A. Christie)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-10 07:00 ` Alexander Lakhin <[email protected]>
2026-04-10 10:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alexander Lakhin @ 2026-04-10 07:00 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello Alvaro,
08.04.2026 00:38, Alvaro Herrera wrote:
> Okay. I implemented that now, and here it is. ...
Could you please look at an assertion failure produced by the following
script, starting from 0d3dba38c:?
createdb db1
createdb db2
echo "
CREATE TABLE t0 (a text);
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
INSERT INTO t0 VALUES ('a');
SELECT pg_sleep(1);
" | psql db1 &
echo "
CREATE TABLE t1 (id int PRIMARY KEY);
CREATE TABLE t2 (id int PRIMARY KEY, a TEXT, FOREIGN KEY (id) REFERENCES t1);
SET min_parallel_table_scan_size = 1;
REPACK (CONCURRENTLY) t2;
" | psql db2
wait
It triggers for me:
TRAP: failed Assert("TransactionIdPrecedesOrEquals(TransactionXmin, RecentXmin)"), File: "procarray.c", Line: 2071, PID:
3529520
postgres: parallel worker for PID 3529517 (ExceptionalCondition+0x69)[0x62f724b19c4c]
postgres: parallel worker for PID 3529517 (+0x522456)[0x62f72498e456]
postgres: parallel worker for PID 3529517 (GetSnapshotData+0x6b)[0x62f72498f50c]
postgres: parallel worker for PID 3529517 (GetNonHistoricCatalogSnapshot+0x4b)[0x62f724b5bd90]
postgres: parallel worker for PID 3529517 (GetCatalogSnapshot+0x20)[0x62f724b5ce7b]
postgres: parallel worker for PID 3529517 (systable_beginscan+0x10b)[0x62f7245c31e7]
postgres: parallel worker for PID 3529517 (+0x69e1e7)[0x62f724b0a1e7]
postgres: parallel worker for PID 3529517 (+0x69e5a6)[0x62f724b0a5a6]
postgres: parallel worker for PID 3529517 (+0x6a4d86)[0x62f724b10d86]
postgres: parallel worker for PID 3529517 (RelationIdGetRelation+0x83)[0x62f724b11208]
postgres: parallel worker for PID 3529517 (relation_open+0x1e)[0x62f72456c235]
...
2026-04-10 05:59:51.471 UTC [3529495] LOG: background worker "parallel worker" (PID 3529520) was terminated by signal
6: Aborted
Best regards,
Alexander
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 07:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
@ 2026-04-10 10:57 ` Antonin Houska <[email protected]>
2026-04-12 14:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-10 10:57 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alexander Lakhin <[email protected]> wrote:
> Could you please look at an assertion failure produced by the following
> script, starting from 0d3dba38c:?
> createdb db1
> createdb db2
>
> echo "
> CREATE TABLE t0 (a text);
> BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
> INSERT INTO t0 VALUES ('a');
> SELECT pg_sleep(1);
> " | psql db1 &
>
> echo "
> CREATE TABLE t1 (id int PRIMARY KEY);
> CREATE TABLE t2 (id int PRIMARY KEY, a TEXT, FOREIGN KEY (id) REFERENCES t1);
> SET min_parallel_table_scan_size = 1;
> REPACK (CONCURRENTLY) t2;
> " | psql db2
> wait
>
> It triggers for me:
> TRAP: failed Assert("TransactionIdPrecedesOrEquals(TransactionXmin, RecentXmin)"), File: "procarray.c", Line: 2071, PID: 3529520
Attached is a fix that works for me.
Nevertheless, REPACK (CONCURRENTLY) in your test goes ahead only due to commit
0d3dba38c7, which will probably be reverted [1]. Then REPACK will wait for the
transaction in the other database (db1) to complete before it can actually
start.
Thanks for the report!
[1] https://www.postgresql.org/message-id/cdgw4sbbfcgk6du3iv54r2dgiy4tfywoklbotlmj4irxavdcr3@glxfw5jj277...
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 07:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
2026-04-10 10:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-12 14:00 ` Alexander Lakhin <[email protected]>
2026-04-13 09:42 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alexander Lakhin @ 2026-04-12 14:00 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello Antonin and Alvaro,
10.04.2026 13:57, Antonin Houska wrote:
> Attached is a fix that works for me.
>
> Nevertheless, REPACK (CONCURRENTLY) in your test goes ahead only due to commit
> 0d3dba38c7, which will probably be reverted [1]. Then REPACK will wait for the
> transaction in the other database (db1) to complete before it can actually
> start.
Thank you for the fix!
I've stumbled upon one more issue with this feature:
CREATE TABLE t (i int PRIMARY KEY);
REPACK (CONCURRENTLY) t;
fails for me with sanitizers enabled and
min_dynamic_shared_memory = '1GB'
in postgresql.conf as below:
2026-04-12 13:23:02.000 UTC [2733633] LOG: statement: REPACK (CONCURRENTLY) t;
repack.c:3373:15: runtime error: load of value 240, which is not a valid value for type '_Bool'
#0 0x6441f7eba454 in start_repack_decoding_worker .../src/backend/commands/repack.c:3373
#1 0x6441f7ebdaad in rebuild_relation .../src/backend/commands/repack.c:1010
#2 0x6441f7ebe9a2 in cluster_rel .../src/backend/commands/repack.c:656
#3 0x6441f7ebefea in process_single_relation .../src/backend/commands/repack.c:2359
#4 0x6441f7ebf870 in ExecRepack .../src/backend/commands/repack.c:296
#5 0x6441f886f20e in standard_ProcessUtility .../src/backend/tcop/utility.c:867
...
2026-04-12 13:23:03.620 UTC [2733620] LOG: client backend (PID 2733633) was terminated by signal 6: Aborted
2026-04-12 13:23:03.620 UTC [2733620] DETAIL: Failed process was running: REPACK (CONCURRENTLY) t;
Could you please have a look?
Best regards,
Alexander
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 07:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
2026-04-10 10:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-12 14:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
@ 2026-04-13 09:42 ` Antonin Houska <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-13 09:42 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alexander Lakhin <[email protected]> wrote:
> I've stumbled upon one more issue with this feature:
> CREATE TABLE t (i int PRIMARY KEY);
> REPACK (CONCURRENTLY) t;
>
> fails for me with sanitizers enabled and
> min_dynamic_shared_memory = '1GB'
> in postgresql.conf as below:
> 2026-04-12 13:23:02.000 UTC [2733633] LOG: statement: REPACK (CONCURRENTLY) t;
> repack.c:3373:15: runtime error: load of value 240, which is not a valid value for type '_Bool'
> #0 0x6441f7eba454 in start_repack_decoding_worker .../src/backend/commands/repack.c:3373
> #1 0x6441f7ebdaad in rebuild_relation .../src/backend/commands/repack.c:1010
> #2 0x6441f7ebe9a2 in cluster_rel .../src/backend/commands/repack.c:656
> #3 0x6441f7ebefea in process_single_relation .../src/backend/commands/repack.c:2359
> #4 0x6441f7ebf870 in ExecRepack .../src/backend/commands/repack.c:296
> #5 0x6441f886f20e in standard_ProcessUtility .../src/backend/tcop/utility.c:867
I could not reproduce the problem, but noticed that the field is not
initialized correctly. Please confirm that 0001 should fix that.
While working on it, I noticed that one field can be removed from
DecodingWorkerShared - 0002 removes that.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-10 10:53 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-21 07:24 ` Re: Adding REPACK [concurrently] Chao Li <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2026-04-10 10:53 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
When testing REPACK concurrently, I noticed that all WALs are retained from
the moment REPACK begins copying data to the new table until the command
finishes replaying concurrent changes on the new table and stops the repack
decoding worker.
I understand the reason: the REPACK command itself starts a long-running
transaction, and logical decoding does not advance restart_lsn beyond the
oldest running transaction's start position. As a result, slot.restart_lsn
remains unchanged, preventing the checkpointer from recycling WALs.
However, since REPACK can run for a long time (hours or even days), I'd like
to confirm whether this is expected behavior or if we plan to improve it
in the future ? And additionally, IIUC, REPACK without using concurrent option
does not have this issue.
Given that we do not restart a REPACK, I think the repack decoding worker
should be able to advance restart_lsn each time after writing changes
(similar to how a physical slot behaves). To illustrate this, I've written
a patch (attached) that implements this approach, and it works fine for me.
BTW, catalog_xmin also won't advance, but that seems not a big issue as
the REPACK transaction itself also holds a snapshot that retains catalog tuples,
so advancing catalog_xmin wouldn't change the situation anyway.
Thoughts ?
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v1-0001-Allow-old-WALs-to-be-removed-during-REPACK-CONCUR.patch (2.3K, 2-v1-0001-Allow-old-WALs-to-be-removed-during-REPACK-CONCUR.patch)
download | inline diff:
From 3b4c9a890fdeb854949a36b6be8f1fc00a5edafe Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Fri, 10 Apr 2026 16:24:55 +0800
Subject: [PATCH v1] Allow old WALs to be removed during REPACK CONCURRENTLY
---
src/backend/commands/repack_worker.c | 14 +++++++++++++-
src/backend/replication/logical/logical.c | 4 +++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
index 5bd020e0184..50f000ca6df 100644
--- a/src/backend/commands/repack_worker.c
+++ b/src/backend/commands/repack_worker.c
@@ -394,12 +394,24 @@ decode_concurrent_changes(LogicalDecodingContext *ctx,
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance.
+ * system that the slot can advance.
+ *
+ * Once REPACK begins copying data to the new table, the logical
+ * decoding machinery prevents the slot from advancing beyond the
+ * oldest running transaction (which is the REPACK transaction
+ * itself). As a result, restart_lsn and catalog_xmin can no longer
+ * advance automatically.
+ *
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed. We do not
+ * advance catalog_xmin here because the REPACK transaction anyway
+ * holds a snapshot that prevents catalog tuple removal.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
if (segno_new != repack_current_segment)
{
+ LogicalIncreaseRestartDecodingForSlot(end_lsn, end_lsn);
LogicalConfirmReceivedLocation(end_lsn);
elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
(uint32) (end_lsn >> 32), (uint32) end_lsn);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index d8e02c53558..baa639835f7 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1913,8 +1913,10 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
}
else
{
--
2.53.0.windows.2
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-04-10 13:21 ` Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-10 13:21 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Zhijie Hou (Fujitsu) <[email protected]> wrote:
> When testing REPACK concurrently, I noticed that all WALs are retained from
> the moment REPACK begins copying data to the new table until the command
> finishes replaying concurrent changes on the new table and stops the repack
> decoding worker.
>
> I understand the reason: the REPACK command itself starts a long-running
> transaction, and logical decoding does not advance restart_lsn beyond the
> oldest running transaction's start position. As a result, slot.restart_lsn
> remains unchanged, preventing the checkpointer from recycling WALs.
I think you're right, sorry for the omission.
> However, since REPACK can run for a long time (hours or even days), I'd like
> to confirm whether this is expected behavior or if we plan to improve it
> in the future ? And additionally,
Yes, it will be improved. I have a draft patch for it, will rebase and post it
soon. The plan is to:
1) preserve the original xmin/xmax of the tuples when we insert them into the
new heap. Thus, besides achieving MVCC safety, we won't need XID assigned for
most of the time.
2) do catalog changes in separate transactions - XID needed here, but these
transactions take very short time.
3) use a single snapshot only for limited number of tuples/pages. When more
data needs to be copied, a new snapshot is built, supposedly with higher
->xmin than the prevous one.
> IIUC, REPACK without using concurrent option does not have this issue.
It does not have the WAL recycling issue because it does not need to read
WAL. However it also runs in a long transaction. Even though it does not need
XID for the actual heap rewriting, it gets one at the moment it locks the
table using AccessExclusiveLock (which is at the very beginning).
> Given that we do not restart a REPACK, I think the repack decoding worker
> should be able to advance restart_lsn each time after writing changes
> (similar to how a physical slot behaves). To illustrate this, I've written
> a patch (attached) that implements this approach, and it works fine for me.
LGTM, thanks!
> BTW, catalog_xmin also won't advance, but that seems not a big issue as
> the REPACK transaction itself also holds a snapshot that retains catalog tuples,
> so advancing catalog_xmin wouldn't change the situation anyway.
The snapshot "resetting" (mentioned above) should fix this problem too.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-25 06:26 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2026-05-25 06:26 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
> -----Original Message-----
> From: Antonin Houska <[email protected]>
> Sent: Friday, April 10, 2026 9:22 PM
> To: Hou, Zhijie/侯 志杰 <[email protected]>
> Cc: Alvaro Herrera <[email protected]>; Amit Kapila
> <[email protected]>; Kuroda, Hayato/黒田 隼人
> <[email protected]>; Srinath Reddy Sadipiralla
> <[email protected]>; Mihail Nikalayeu <[email protected]>;
> Matthias van de Meent <[email protected]>; Pg Hackers
> <[email protected]>; Robert Treat <[email protected]>
> Subject: Re: Adding REPACK [concurrently]
>
> Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
> > When testing REPACK concurrently, I noticed that all WALs are retained
> > from the moment REPACK begins copying data to the new table until the
> > command finishes replaying concurrent changes on the new table and
> > stops the repack decoding worker.
> >
> > I understand the reason: the REPACK command itself starts a
> > long-running transaction, and logical decoding does not advance
> > restart_lsn beyond the oldest running transaction's start position. As
> > a result, slot.restart_lsn remains unchanged, preventing the checkpointer
> from recycling WALs.
>
> I think you're right, sorry for the omission.
>
> > IIUC, REPACK without using concurrent option does not have this issue.
>
> It does not have the WAL recycling issue because it does not need to read
> WAL. However it also runs in a long transaction. Even though it does not need
> XID for the actual heap rewriting, it gets one at the moment it locks the table
> using AccessExclusiveLock (which is at the very beginning).
>
> > Given that we do not restart a REPACK, I think the repack decoding
> > worker should be able to advance restart_lsn each time after writing
> > changes (similar to how a physical slot behaves). To illustrate this,
> > I've written a patch (attached) that implements this approach, and it works
> fine for me.
>
> LGTM, thanks!
>
Thanks for reviewing!
After listening to the REPACK talk at pgconf.dev this year, I understand that
WAL accumulation during REPACK CONCURRENTLY is not intended behavior. I think we
can consider fixing this in the current release. Attached is the rebased
patch, with comments adjusted based on Chao Li's comments.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v2-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch (3.0K, 2-v2-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch)
download | inline diff:
From 87ba918b9020ce9fa0d4852f222221f521d8701f Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Fri, 10 Apr 2026 16:24:55 +0800
Subject: [PATCH v2] Allow old WAL recycling during REPACK CONCURRENTLY
During REPACK CONCURRENTLY, logical decoding can keep replication
slot.restart_lsn pinned behind the oldest running transaction, which is often
the long-lived REPACK transaction itself. As a result, old WAL segments are
retained longer than necessary.
This commit advances the replication slot each time WAL insertion crosses a
segment boundary, so obsolete WAL files can be recycled while REPACK is still
running.
This change does not advance catalog_xmin. REPACK already holds a snapshot that
prevents catalog dead tuple removal, so catalog_xmin handling can be addressed
independently.
---
src/backend/commands/repack_worker.c | 14 +++++++++++++-
src/backend/replication/logical/logical.c | 8 +++++++-
2 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
index b84041372b8..eed69d36508 100644
--- a/src/backend/commands/repack_worker.c
+++ b/src/backend/commands/repack_worker.c
@@ -397,12 +397,24 @@ decode_concurrent_changes(LogicalDecodingContext *ctx,
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance.
+ * system that the slot can advance.
+ *
+ * Once REPACK begins copying data to the new table, the logical
+ * decoding machinery prevents the slot from advancing beyond the
+ * oldest running transaction (which is the REPACK transaction
+ * itself). As a result, restart_lsn and catalog_xmin can no
+ * longer advance automatically.
+ *
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed. We do not
+ * advance catalog_xmin here because the REPACK transaction anyway
+ * holds a snapshot that prevents catalog dead tuple removal.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
if (segno_new != repack_current_segment)
{
+ LogicalIncreaseRestartDecodingForSlot(end_lsn, end_lsn);
LogicalConfirmReceivedLocation(end_lsn);
elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
(uint32) (end_lsn >> 32), (uint32) end_lsn);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index a33a685dcc6..11b64de3d90 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1913,8 +1913,14 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ /*
+ * Now the new restart_lsn is safely on disk, recompute the global WAL
+ * retention requirement.
+ */
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
}
else
{
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-05-26 15:31 ` Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-26 15:31 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-25, Zhijie Hou (Fujitsu) wrote:
> After listening to the REPACK talk at pgconf.dev this year, I understand that
> WAL accumulation during REPACK CONCURRENTLY is not intended behavior. I think we
> can consider fixing this in the current release. Attached is the rebased
> patch, with comments adjusted based on Chao Li's comments.
You're right, this is a thinko. I'll look at your patch hoping to get
it pushed shortly. I wonder if we should add a TAP test to verify that
WAL files are actually removed? Sounds a bit excessive TBH, but maybe
it isn't really.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I think my standards have lowered enough that now I think 'good design'
is when the page doesn't irritate the living f*ck out of me." (JWZ)
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-27 08:08 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-28 00:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-29 22:25 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2026-05-27 08:08 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tuesday, May 26, 2026 11:32 PM Alvaro Herrera <[email protected]> wrote:
> On 2026-May-25, Zhijie Hou (Fujitsu) wrote:
>
> > After listening to the REPACK talk at pgconf.dev this year, I
> > understand that WAL accumulation during REPACK CONCURRENTLY is not
> > intended behavior. I think we can consider fixing this in the current
> > release. Attached is the rebased patch, with comments adjusted based on
> Chao Li's comments.
>
> You're right, this is a thinko. I'll look at your patch hoping to get it pushed
> shortly. I wonder if we should add a TAP test to verify that WAL files are
> actually removed? Sounds a bit excessive TBH, but maybe it isn't really.
I tried a bit, and the test complexity and speed (< 1s) appear to be within
acceptable limits. I'm attaching 0002 as a reference test.
0001 remains unchanged.
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v3-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch (3.0K, 2-v3-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch)
download | inline diff:
From 698d6e2e475669ffce16e6b4de04b836130e0ed4 Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Fri, 10 Apr 2026 16:24:55 +0800
Subject: [PATCH v3 1/2] Allow old WAL recycling during REPACK CONCURRENTLY
During REPACK CONCURRENTLY, logical decoding can keep replication
slot.restart_lsn pinned behind the oldest running transaction, which is often
the long-lived REPACK transaction itself. As a result, old WAL segments are
retained longer than necessary.
This commit advances the replication slot each time WAL insertion crosses a
segment boundary, so obsolete WAL files can be recycled while REPACK is still
running.
This change does not advance catalog_xmin. REPACK already holds a snapshot that
prevents catalog dead tuple removal, so catalog_xmin handling can be addressed
independently.
---
src/backend/commands/repack_worker.c | 14 +++++++++++++-
src/backend/replication/logical/logical.c | 8 +++++++-
2 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
index b84041372b8..eed69d36508 100644
--- a/src/backend/commands/repack_worker.c
+++ b/src/backend/commands/repack_worker.c
@@ -397,12 +397,24 @@ decode_concurrent_changes(LogicalDecodingContext *ctx,
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance.
+ * system that the slot can advance.
+ *
+ * Once REPACK begins copying data to the new table, the logical
+ * decoding machinery prevents the slot from advancing beyond the
+ * oldest running transaction (which is the REPACK transaction
+ * itself). As a result, restart_lsn and catalog_xmin can no
+ * longer advance automatically.
+ *
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed. We do not
+ * advance catalog_xmin here because the REPACK transaction anyway
+ * holds a snapshot that prevents catalog dead tuple removal.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
if (segno_new != repack_current_segment)
{
+ LogicalIncreaseRestartDecodingForSlot(end_lsn, end_lsn);
LogicalConfirmReceivedLocation(end_lsn);
elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
(uint32) (end_lsn >> 32), (uint32) end_lsn);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b969caae72e..8b8095bd5d8 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1910,8 +1910,14 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ /*
+ * Now the new restart_lsn is safely on disk, recompute the global WAL
+ * retention requirement.
+ */
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
}
else
{
--
2.43.0
[application/octet-stream] v3-0002-Add-a-test-for-repack-concurrently.patch (3.2K, 3-v3-0002-Add-a-test-for-repack-concurrently.patch)
download | inline diff:
From d97df7c2ecf815907b7e14322f6f8fde09951b9b Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Wed, 27 May 2026 15:58:05 +0800
Subject: [PATCH v3 2/2] Add a test for repack concurrently
---
.../recovery/t/046_checkpoint_logical_slot.pl | 74 +++++++++++++++++++
1 file changed, 74 insertions(+)
diff --git a/src/test/recovery/t/046_checkpoint_logical_slot.pl b/src/test/recovery/t/046_checkpoint_logical_slot.pl
index 66761bf56c1..aa32859dd15 100644
--- a/src/test/recovery/t/046_checkpoint_logical_slot.pl
+++ b/src/test/recovery/t/046_checkpoint_logical_slot.pl
@@ -226,4 +226,78 @@ is( $standby->safe_psql(
"t",
'logical slot is not invalidated');
+# Verify that the REPACK slot's restart_lsn can advance while REPACK
+# CONCURRENTLY is still running, allowing WAL files to be recycled during this
+# period.
+
+# Create the table to be repacked and populate it with some data.
+$node->safe_psql(
+ 'postgres',
+ q{
+CREATE TABLE repack_test(i int PRIMARY KEY, t text);
+INSERT INTO repack_test
+SELECT g, md5(g::text)
+FROM generate_series(1, 100) g;
+});
+
+# Pause the REPACK command in the middle of its execution so that the decoding
+# worker continues running, allowing us to test slot restart_lsn advancement
+# later.
+$node->safe_psql('postgres',
+ q(select injection_points_attach('repack-concurrently-before-lock','wait'))
+);
+
+my $repack = $node->background_psql('postgres');
+$repack->query_until(
+ qr/repack_started/,
+ q(
+\echo repack_started
+REPACK (CONCURRENTLY) repack_test;
+\q
+));
+
+# Wait until REPACK reaches the injection point.
+$node->wait_for_event('client backend', 'repack-concurrently-before-lock');
+
+my $restart_lsn_before = $node->safe_psql('postgres',
+ "SELECT restart_lsn FROM pg_replication_slots WHERE slot_name ~ '^repack_[0-9]+' AND slot_type = 'logical' AND temporary;");
+
+# Verify that the replication slot created by the subscription exists and has a
+# valid restart_lsn.
+ok(defined($restart_lsn_before) && $restart_lsn_before ne '',
+ 'REPACK slot has restart_lsn');
+
+my $segment_before = $node->safe_psql('postgres',
+ "SELECT pg_walfile_name('$restart_lsn_before')");
+my $segment_before_path = $node->data_dir . "/pg_wal/$segment_before";
+ok(-f $segment_before_path,
+ "segment for initial restart_lsn exists: $segment_before");
+
+# Switch WAL file on the primary while REPACK is still running and then force
+# WAL removal/recycling with a checkpoint.
+$node->advance_wal(1);
+
+# Wait until the REPACK slot's restart_lsn advances
+ok( $node->poll_query_until(
+ 'postgres', qq[
+ SELECT count(*) > 0
+ FROM pg_replication_slots
+ WHERE slot_name ~ '^repack_[0-9]+'
+ AND slot_type = 'logical'
+ AND temporary
+ AND restart_lsn IS NOT NULL
+ AND restart_lsn <> '$restart_lsn_before'::pg_lsn]),
+ 'REPACK slot restart_lsn advances while command is still running');
+
+$node->safe_psql('postgres', 'CHECKPOINT');
+
+# Test that the old WAL segment was recycled
+ok(!-f $segment_before_path,
+ 'old WAL segment was recycled while REPACK CONCURRENTLY was running');
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('repack-concurrently-before-lock')");
+
+$repack->quit;
+
done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-05-28 00:31 ` Amit Kapila <[email protected]>
2026-05-28 03:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-28 00:31 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, May 27, 2026 at 1:08 AM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> 0001 remains unchanged.
>
Few minor comments:
=================
*
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed.
This is safe only because REPACK creates a temporary slot that is
dropped if REPACK fails — there's no scenario where this slot needs to
restart decoding from
an earlier position while still alive. I feel that is worth a mention.
*
@@ -1910,8 +1910,14 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ /*
+ * Now the new restart_lsn is safely on disk, recompute the global WAL
+ * retention requirement.
+ */
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
This change is not related to this patch, rather we need it even
without this patch, is it worth mentioning in the commit message?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-28 00:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-28 03:34 ` Amit Kapila <[email protected]>
2026-05-28 05:18 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-28 03:34 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, May 27, 2026 at 5:31 PM Amit Kapila <[email protected]> wrote:
>
> On Wed, May 27, 2026 at 1:08 AM Zhijie Hou (Fujitsu)
> <[email protected]> wrote:
> >
> > 0001 remains unchanged.
> >
>
> Few minor comments:
> =================
Commit message says: "This change does not advance catalog_xmin.
REPACK already holds a snapshot that prevents catalog dead tuple
removal, so catalog_xmin handling can be addressed independently.".
Isn't it equally important to advance this, otherwise, for long
running REPACKs dead tuples will be accumulated needlessly? If so, do
we have any ideas to avoid this?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-28 00:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-28 03:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-28 05:18 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-29 15:39 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Zhijie Hou (Fujitsu) @ 2026-05-28 05:18 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Thursday, May 28, 2026 11:34 AM Amit Kapila <[email protected]> wrote:
> On Wed, May 27, 2026 at 5:31 PM Amit Kapila <[email protected]>
> wrote:
> >
> > On Wed, May 27, 2026 at 1:08 AM Zhijie Hou (Fujitsu)
> > <[email protected]> wrote:
> > >
> > > 0001 remains unchanged.
> > >
> >
> > Few minor comments:
> > =================
>
> Commit message says: "This change does not advance catalog_xmin.
> REPACK already holds a snapshot that prevents catalog dead tuple removal,
> so catalog_xmin handling can be addressed independently.".
> Isn't it equally important to advance this, otherwise, for long running REPACKs
> dead tuples will be accumulated needlessly? If so, do we have any ideas to
> avoid this?
My understanding is that dead tuple accumulation is common to all long-running
commands (including CLUSTER, VACUUM FULL, and REPACK without CONCURRENTLY). As
long as a command holds a snapshot for a long time while scanning and copying
data, the backend xmin will cause similar accumulation. So, this doesn't seem
like a new issue to me, and given that catalog_xmin only affect tuples in system
catalog which is less harmful, I thought it could be handled independently.
There was a proposal to improve this case in [1]. Sorry if I've missed something.
Attaching the v4 patch which improved the comments and commit message as
suggested.
[1] https://www.postgresql.org/message-id/125085.1775827305%40localhost
Best Regards,
Hou zj
Attachments:
[application/octet-stream] v4-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch (3.6K, 2-v4-0001-Allow-old-WAL-recycling-during-REPACK-CONCURRENTL.patch)
download | inline diff:
From 74881e1bf03da8a4772b3bc5a24542b6dfb51042 Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Fri, 10 Apr 2026 16:24:55 +0800
Subject: [PATCH v4 1/2] Allow old WAL recycling during REPACK CONCURRENTLY
During REPACK CONCURRENTLY, logical decoding can keep replication
slot.restart_lsn pinned behind the oldest running transaction, which is often
the long-lived REPACK transaction itself. As a result, old WAL segments are
retained longer than necessary.
This commit advances the replication slot each time WAL insertion crosses a
segment boundary, so obsolete WAL files can be recycled while REPACK is still
running.
This change does not advance catalog_xmin. REPACK already holds a snapshot that
prevents catalog dead tuple removal, so catalog_xmin handling can be addressed
independently.
Additionally, this commit improves LogicalConfirmReceivedLocation to compute the
oldest restart LSN whenever slot.restart_lsn is updated. Previously, this
function performed the computation only when catalog_xmin was updated, which was
less problematic because catalog_xmin typically advances in most replication
cases, but not for REPACK.
---
src/backend/commands/repack_worker.c | 20 +++++++++++++++++++-
src/backend/replication/logical/logical.c | 8 +++++++-
2 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
index 4f82eb46bec..56974cbc1f5 100644
--- a/src/backend/commands/repack_worker.c
+++ b/src/backend/commands/repack_worker.c
@@ -397,12 +397,30 @@ decode_concurrent_changes(LogicalDecodingContext *ctx,
/*
* If WAL segment boundary has been crossed, inform the decoding
- * system that the catalog_xmin can advance.
+ * system that the slot can advance.
+ *
+ * Once REPACK begins copying data to the new table, the logical
+ * decoding machinery prevents the slot from advancing beyond the
+ * oldest running transaction (which is the REPACK transaction
+ * itself). As a result, restart_lsn and catalog_xmin can no
+ * longer advance automatically.
+ *
+ * To allow old WAL files to be recycled, we manually advance the
+ * slot each time a WAL segment boundary is crossed. This is safe
+ * because the REPACK slot is temporary and will be dropped
+ * automatically if the REPACK command fails. There is no scenario
+ * where this slot needs to restart decoding from an earlier
+ * position while still alive.
+ *
+ * We do not advance catalog_xmin here because the REPACK
+ * transaction anyway holds a snapshot that prevents catalog dead
+ * tuple removal.
*/
end_lsn = ctx->reader->EndRecPtr;
XLByteToSeg(end_lsn, segno_new, wal_segment_size);
if (segno_new != repack_current_segment)
{
+ LogicalIncreaseRestartDecodingForSlot(end_lsn, end_lsn);
LogicalConfirmReceivedLocation(end_lsn);
elog(DEBUG1, "REPACK: confirmed receive location %X/%X",
(uint32) (end_lsn >> 32), (uint32) end_lsn);
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b969caae72e..8b8095bd5d8 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1910,8 +1910,14 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockRelease(&MyReplicationSlot->mutex);
ReplicationSlotsComputeRequiredXmin(false);
- ReplicationSlotsComputeRequiredLSN();
}
+
+ /*
+ * Now the new restart_lsn is safely on disk, recompute the global WAL
+ * retention requirement.
+ */
+ if (updated_restart)
+ ReplicationSlotsComputeRequiredLSN();
}
else
{
--
2.43.0
[application/octet-stream] v4-0002-Add-a-test-for-repack-concurrently.patch (3.2K, 3-v4-0002-Add-a-test-for-repack-concurrently.patch)
download | inline diff:
From 3a82ef6b68280efb1ed008cf436b5a33eed1488f Mon Sep 17 00:00:00 2001
From: Zhijie Hou <[email protected]>
Date: Wed, 27 May 2026 15:58:05 +0800
Subject: [PATCH v4 2/2] Add a test for repack concurrently
---
.../recovery/t/046_checkpoint_logical_slot.pl | 74 +++++++++++++++++++
1 file changed, 74 insertions(+)
diff --git a/src/test/recovery/t/046_checkpoint_logical_slot.pl b/src/test/recovery/t/046_checkpoint_logical_slot.pl
index 66761bf56c1..aa32859dd15 100644
--- a/src/test/recovery/t/046_checkpoint_logical_slot.pl
+++ b/src/test/recovery/t/046_checkpoint_logical_slot.pl
@@ -226,4 +226,78 @@ is( $standby->safe_psql(
"t",
'logical slot is not invalidated');
+# Verify that the REPACK slot's restart_lsn can advance while REPACK
+# CONCURRENTLY is still running, allowing WAL files to be recycled during this
+# period.
+
+# Create the table to be repacked and populate it with some data.
+$node->safe_psql(
+ 'postgres',
+ q{
+CREATE TABLE repack_test(i int PRIMARY KEY, t text);
+INSERT INTO repack_test
+SELECT g, md5(g::text)
+FROM generate_series(1, 100) g;
+});
+
+# Pause the REPACK command in the middle of its execution so that the decoding
+# worker continues running, allowing us to test slot restart_lsn advancement
+# later.
+$node->safe_psql('postgres',
+ q(select injection_points_attach('repack-concurrently-before-lock','wait'))
+);
+
+my $repack = $node->background_psql('postgres');
+$repack->query_until(
+ qr/repack_started/,
+ q(
+\echo repack_started
+REPACK (CONCURRENTLY) repack_test;
+\q
+));
+
+# Wait until REPACK reaches the injection point.
+$node->wait_for_event('client backend', 'repack-concurrently-before-lock');
+
+my $restart_lsn_before = $node->safe_psql('postgres',
+ "SELECT restart_lsn FROM pg_replication_slots WHERE slot_name ~ '^repack_[0-9]+' AND slot_type = 'logical' AND temporary;");
+
+# Verify that the replication slot created by the subscription exists and has a
+# valid restart_lsn.
+ok(defined($restart_lsn_before) && $restart_lsn_before ne '',
+ 'REPACK slot has restart_lsn');
+
+my $segment_before = $node->safe_psql('postgres',
+ "SELECT pg_walfile_name('$restart_lsn_before')");
+my $segment_before_path = $node->data_dir . "/pg_wal/$segment_before";
+ok(-f $segment_before_path,
+ "segment for initial restart_lsn exists: $segment_before");
+
+# Switch WAL file on the primary while REPACK is still running and then force
+# WAL removal/recycling with a checkpoint.
+$node->advance_wal(1);
+
+# Wait until the REPACK slot's restart_lsn advances
+ok( $node->poll_query_until(
+ 'postgres', qq[
+ SELECT count(*) > 0
+ FROM pg_replication_slots
+ WHERE slot_name ~ '^repack_[0-9]+'
+ AND slot_type = 'logical'
+ AND temporary
+ AND restart_lsn IS NOT NULL
+ AND restart_lsn <> '$restart_lsn_before'::pg_lsn]),
+ 'REPACK slot restart_lsn advances while command is still running');
+
+$node->safe_psql('postgres', 'CHECKPOINT');
+
+# Test that the old WAL segment was recycled
+ok(!-f $segment_before_path,
+ 'old WAL segment was recycled while REPACK CONCURRENTLY was running');
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('repack-concurrently-before-lock')");
+
+$repack->quit;
+
done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-28 00:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-28 03:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-28 05:18 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-05-29 15:39 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Amit Kapila @ 2026-05-29 15:39 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, May 27, 2026 at 10:18 PM Zhijie Hou (Fujitsu)
<[email protected]> wrote:
>
> On Thursday, May 28, 2026 11:34 AM Amit Kapila <[email protected]> wrote:
> > On Wed, May 27, 2026 at 5:31 PM Amit Kapila <[email protected]>
> > wrote:
> > >
> > > On Wed, May 27, 2026 at 1:08 AM Zhijie Hou (Fujitsu)
> > > <[email protected]> wrote:
> > > >
> > > > 0001 remains unchanged.
> > > >
> > >
> > > Few minor comments:
> > > =================
> >
> > Commit message says: "This change does not advance catalog_xmin.
> > REPACK already holds a snapshot that prevents catalog dead tuple removal,
> > so catalog_xmin handling can be addressed independently.".
> > Isn't it equally important to advance this, otherwise, for long running REPACKs
> > dead tuples will be accumulated needlessly? If so, do we have any ideas to
> > avoid this?
>
> My understanding is that dead tuple accumulation is common to all long-running
> commands (including CLUSTER, VACUUM FULL, and REPACK without CONCURRENTLY). As
> long as a command holds a snapshot for a long time while scanning and copying
> data, the backend xmin will cause similar accumulation. So, this doesn't seem
> like a new issue to me, and given that catalog_xmin only affect tuples in system
> catalog which is less harmful, I thought it could be handled independently.
> There was a proposal to improve this case in [1].
>
Fair enough. It makes sense to deal with catalog_xmin separately.
> Attaching the v4 patch which improved the comments and commit message as
> suggested.
>
I haven't tested it but otherwise the code changes looks good to me.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-25 06:26 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-05-29 22:25 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-29 22:25 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-27, Zhijie Hou (Fujitsu) wrote:
> I tried a bit, and the test complexity and speed (< 1s) appear to be within
> acceptable limits. I'm attaching 0002 as a reference test.
>
> 0001 remains unchanged.
Pushed 0001, in two parts. I rewrote the comment in
decode_concurrent_changes() though, hope it ended up okay.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Los cuentos de hadas no dan al niño su primera idea sobre los monstruos.
Lo que le dan es su primera idea de la posible derrota del monstruo."
(G. K. Chesterton)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 21:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-10 10:53 ` RE: Adding REPACK [concurrently] Zhijie Hou (Fujitsu) <[email protected]>
@ 2026-04-21 07:24 ` Chao Li <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Chao Li @ 2026-04-21 07:24 UTC (permalink / raw)
To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
> On Apr 10, 2026, at 18:53, Zhijie Hou (Fujitsu) <[email protected]> wrote:
>
> Hi,
>
> When testing REPACK concurrently, I noticed that all WALs are retained from
> the moment REPACK begins copying data to the new table until the command
> finishes replaying concurrent changes on the new table and stops the repack
> decoding worker.
>
> I understand the reason: the REPACK command itself starts a long-running
> transaction, and logical decoding does not advance restart_lsn beyond the
> oldest running transaction's start position. As a result, slot.restart_lsn
> remains unchanged, preventing the checkpointer from recycling WALs.
>
> However, since REPACK can run for a long time (hours or even days), I'd like
> to confirm whether this is expected behavior or if we plan to improve it
> in the future ? And additionally, IIUC, REPACK without using concurrent option
> does not have this issue.
>
> Given that we do not restart a REPACK, I think the repack decoding worker
> should be able to advance restart_lsn each time after writing changes
> (similar to how a physical slot behaves). To illustrate this, I've written
> a patch (attached) that implements this approach, and it works fine for me.
>
> BTW, catalog_xmin also won't advance, but that seems not a big issue as
> the REPACK transaction itself also holds a snapshot that retains catalog tuples,
> so advancing catalog_xmin wouldn't change the situation anyway.
>
> Thoughts ?
>
> Best Regards,
> Hou zj
> <v1-0001-Allow-old-WALs-to-be-removed-during-REPACK-CONCUR.patch>
I found the same problem with LogicalConfirmReceivedLocation and posted a fix in a separate thread [1]. So I would withdraw my patch.
Looking at this patch, the change is exactly the same as what I did in [1], but I think the code comment should be updated as well. For the comment change, please see my patch in [1].
[1] https://www.postgresql.org/message-id/D8D9F770-DAA2-482C-A7E0-F87E5104C13E%40gmail.com
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-04 13:16 ` Srinath Reddy Sadipiralla <[email protected]>
2026-04-04 13:55 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-04-04 13:16 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi Alvaro,
On Sat, Apr 4, 2026 at 1:01 AM Alvaro Herrera <[email protected]>
wrote:
> On 2026-Apr-03, Alvaro Herrera wrote:
>
> > - I polished the patch to reserve replication slots for REPACK. Given
> > the new implementation of 0006 that was submitted implies that we can
> > now run multiple repacks concurrently, I changed the default of 1 to 5.
>
> Srinath let me know that this new part was causing CI failures on
> Windows. This version v51 should be okay (or, at least, it passes for
> me on CI).
yes indeed , TotalMaxReplicationSlots was the culprit , as the window
after getting repack worked was forked , i think this was set back to 0.
Thanks for fixing this with "repack" flag , initially i thought why can't we
go with a macro
#define TotalMaxReplicationSlots (max_replication_slots +
max_repack_replication_slots)
but i thought maybe in future we might need this "repack" flag to add
more slot requirements.
--
Thanks,
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 14:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 13:16 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
@ 2026-04-04 13:55 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-04 13:55 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-04, Srinath Reddy Sadipiralla wrote:
> yes indeed , TotalMaxReplicationSlots was the culprit , as the window
> after getting repack worked was forked , i think this was set back to 0.
> Thanks for fixing this with "repack" flag , initially i thought why can't we
> go with a macro
> #define TotalMaxReplicationSlots (max_replication_slots +
> max_repack_replication_slots)
> but i thought maybe in future we might need this "repack" flag to add
> more slot requirements.
Yeah, the other option would have been to add the variable to
restore_backend_variables() so that it's restored after the fork+exec,
but that didn't seem better.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Niemand ist mehr Sklave, als der sich für frei hält, ohne es zu sein."
Nadie está tan esclavizado como el que se cree libre no siéndolo
(Johann Wolfgang von Goethe)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-03 17:24 ` Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-03 17:24 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-03, Antonin Houska wrote:
> diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
> index 00b21ede481..c25dbeadff3 100644
> --- a/src/backend/commands/repack_worker.c
> +++ b/src/backend/commands/repack_worker.c
> @@ -233,6 +234,13 @@ repack_setup_logical_decoding(Oid relid)
>
> EnsureLogicalDecodingEnabled();
>
> + /*
> + * By declaring that our output plugin does not need shared catalogs, we
> + * avoid waiting for completion of transactions running in other databases
> + * than the one we're connected to.
> + */
> + accessSharedCatalogsInDecoding = false;
> +
> /*
> * Neither prepare_write nor do_write callback nor update_progress is
> * useful for us.
I find this reliance on a global variable for this a bit icky. Would it
work to instead change the CreateInitDecodingContext() signature, so
that instead of "bool need_full_snapshot" it has a three-valued boolean
to distinguish the two cases from the original plus this new one? I
think the value could be stored in LogicalDecodingContext, from where
standby_decode() could obtain it.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"The important things in the world are problems with society that we don't
understand at all. The machines will become more complicated but they won't
be more complicated than the societies that run them." (Freeman Dyson)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-04 08:50 ` Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-04 08:50 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-03, Antonin Houska wrote:
>
> > diff --git a/src/backend/commands/repack_worker.c b/src/backend/commands/repack_worker.c
> > index 00b21ede481..c25dbeadff3 100644
> > --- a/src/backend/commands/repack_worker.c
> > +++ b/src/backend/commands/repack_worker.c
>
> > @@ -233,6 +234,13 @@ repack_setup_logical_decoding(Oid relid)
> >
> > EnsureLogicalDecodingEnabled();
> >
> > + /*
> > + * By declaring that our output plugin does not need shared catalogs, we
> > + * avoid waiting for completion of transactions running in other databases
> > + * than the one we're connected to.
> > + */
> > + accessSharedCatalogsInDecoding = false;
> > +
> > /*
> > * Neither prepare_write nor do_write callback nor update_progress is
> > * useful for us.
>
> I find this reliance on a global variable for this a bit icky. Would it
> work to instead change the CreateInitDecodingContext() signature, so
> that instead of "bool need_full_snapshot" it has a three-valued boolean
> to distinguish the two cases from the original plus this new one? I
> think the value could be stored in LogicalDecodingContext, from where
> standby_decode() could obtain it.
I agree that the global variable is not handy, but instead of modifying
CreateInitDecodingContext(), how about adding a boolean returning callback to
OutputPluginCallbacks? The point is that whether shared catalogs are needed
during the decoding or not is actually property of the plugin.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-04 09:48 ` Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-04 09:48 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-04, Antonin Houska wrote:
> I agree that the global variable is not handy, but instead of modifying
> CreateInitDecodingContext(), how about adding a boolean returning callback to
> OutputPluginCallbacks? The point is that whether shared catalogs are needed
> during the decoding or not is actually property of the plugin.
Oh, yeah, that sounds good to me.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"After a quick R of TFM, all I can say is HOLY CR** THAT IS COOL! PostgreSQL was
amazing when I first started using it at 7.2, and I'm continually astounded by
learning new features and techniques made available by the continuing work of
the development team."
Berend Tober, http://archives.postgresql.org/pgsql-hackers/2007-08/msg01009.php
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-04 15:29 ` Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-04 15:29 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-04, Antonin Houska wrote:
>
> > I agree that the global variable is not handy, but instead of modifying
> > CreateInitDecodingContext(), how about adding a boolean returning callback to
> > OutputPluginCallbacks? The point is that whether shared catalogs are needed
> > during the decoding or not is actually property of the plugin.
>
> Oh, yeah, that sounds good to me.
This is it. New callback was actually not needed, I just added a new flag to
the OutputPluginOptions structure.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-04 23:53 ` Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-04 23:53 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-04, Antonin Houska wrote:
> Alvaro Herrera <[email protected]> wrote:
>
> > On 2026-Apr-04, Antonin Houska wrote:
> >
> > > I agree that the global variable is not handy, but instead of modifying
> > > CreateInitDecodingContext(), how about adding a boolean returning callback to
> > > OutputPluginCallbacks? The point is that whether shared catalogs are needed
> > > during the decoding or not is actually property of the plugin.
> >
> > Oh, yeah, that sounds good to me.
>
> This is it. New callback was actually not needed, I just added a new flag to
> the OutputPluginOptions structure.
Thank you, I removed the previous one and picked up this one (it's 0001
here.) The only potentially troublesome thing I see with it is this change:
/*
* Update range of interesting xids based on the running xacts
* information. We don't increase ->xmax using it, because once we are in
* a consistent state we can do that ourselves and much more efficiently
* so, because we only need to do it for catalog transactions since we
* only ever look at those.
*
* NB: We only increase xmax when a catalog modifying transaction commits
* (see SnapBuildCommitTxn). Because of this, xmax can be lower than
* xmin, which looks odd but is correct and actually more efficient, since
* we hit fast paths in heapam_visibility.c.
+ *
+ * If database specific transaction info was used during startup, the info
+ * for the whole cluster can make xmin go backwards. That would be bad
+ * because we might no longer have older XIDs in ->committed.
*/
- builder->xmin = running->oldestRunningXid;
+ if (NormalTransactionIdFollows(running->oldestRunningXid, builder->xmin))
+ builder->xmin = running->oldestRunningXid;
I can't see any problem with advancing the ->xmin only when it goes
forward, but I wonder if it's possible to introduce any bugs this way.
This bit looks funny though:
/*
* Advance the xmin limit for the current replication slot, to allow
* vacuum to clean up the tuples this slot has been protecting.
*
* The reorderbuffer might have an xmin among the currently running
* snapshots; use it if so. If not, we need only consider the snapshots
* we'll produce later, which can't be less than the oldest running xid in
* the record we're reading now.
*/
xmin = ReorderBufferGetOldestXmin(builder->reorder);
- if (xmin == InvalidTransactionId)
+ /*
+ * Like above, do not let slot xmin go backwards.
+ */
+ if (xmin == InvalidTransactionId && !db_specific)
xmin = running->oldestRunningXid;
I probably need some sleep, but this doesn't make sense to me.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"No me acuerdo, pero no es cierto. No es cierto, y si fuera cierto,
no me acuerdo." (Augusto Pinochet a una corte de justicia)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-05 07:54 ` Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-05 07:54 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-04, Antonin Houska wrote:
>
> > Alvaro Herrera <[email protected]> wrote:
> >
> > > On 2026-Apr-04, Antonin Houska wrote:
> > >
> > > > I agree that the global variable is not handy, but instead of modifying
> > > > CreateInitDecodingContext(), how about adding a boolean returning callback to
> > > > OutputPluginCallbacks? The point is that whether shared catalogs are needed
> > > > during the decoding or not is actually property of the plugin.
> > >
> > > Oh, yeah, that sounds good to me.
> >
> > This is it. New callback was actually not needed, I just added a new flag to
> > the OutputPluginOptions structure.
>
> Thank you, I removed the previous one and picked up this one (it's 0001
> here.) The only potentially troublesome thing I see with it is this change:
>
> /*
> * Update range of interesting xids based on the running xacts
> * information. We don't increase ->xmax using it, because once we are in
> * a consistent state we can do that ourselves and much more efficiently
> * so, because we only need to do it for catalog transactions since we
> * only ever look at those.
> *
> * NB: We only increase xmax when a catalog modifying transaction commits
> * (see SnapBuildCommitTxn). Because of this, xmax can be lower than
> * xmin, which looks odd but is correct and actually more efficient, since
> * we hit fast paths in heapam_visibility.c.
> + *
> + * If database specific transaction info was used during startup, the info
> + * for the whole cluster can make xmin go backwards. That would be bad
> + * because we might no longer have older XIDs in ->committed.
> */
> - builder->xmin = running->oldestRunningXid;
> + if (NormalTransactionIdFollows(running->oldestRunningXid, builder->xmin))
> + builder->xmin = running->oldestRunningXid;
>
>
> I can't see any problem with advancing the ->xmin only when it goes
> forward, but I wonder if it's possible to introduce any bugs this way.
>
>
> This bit looks funny though:
>
> /*
> * Advance the xmin limit for the current replication slot, to allow
> * vacuum to clean up the tuples this slot has been protecting.
> *
> * The reorderbuffer might have an xmin among the currently running
> * snapshots; use it if so. If not, we need only consider the snapshots
> * we'll produce later, which can't be less than the oldest running xid in
> * the record we're reading now.
> */
> xmin = ReorderBufferGetOldestXmin(builder->reorder);
> - if (xmin == InvalidTransactionId)
> + /*
> + * Like above, do not let slot xmin go backwards.
> + */
> + if (xmin == InvalidTransactionId && !db_specific)
> xmin = running->oldestRunningXid;
>
> I probably need some sleep, but this doesn't make sense to me.
ok, maybe just skip the whole cleanup in that special case.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-05 11:50 ` Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-05 11:50 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-05, Antonin Houska wrote:
> ok, maybe just skip the whole cleanup in that special case.
Hmm, should we make this test only in the db_specific case? Doing it
unconditionally makes me a bit nervous (maybe because I don't fully
understand historic snapshot building).
Anyway I just pushed the addition of a progress-suppression bit to
index_create() interface. Here's the rest of the series.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Ninguna manada de bestias tiene una voz tan horrible como la humana" (Orual)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-05 16:30 ` Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-05 16:30 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-05, Antonin Houska wrote:
>
> > ok, maybe just skip the whole cleanup in that special case.
>
> Hmm, should we make this test only in the db_specific case? Doing it
> unconditionally makes me a bit nervous (maybe because I don't fully
> understand historic snapshot building).
I thought about adding Assert(db_specific) in front of the new return
statement. So what you suggest makes sense to me.
As far as I understand, the xl_running_xacts record is not directly involved
in the snapshot build. Rather, the list of XIDs for snapshots is created and
updated by processing COMMIT and ABORT records.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-05 18:56 ` Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:03 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-05 18:56 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
So I've been trying to understand the "Introduce an option to make
logical replication database specific." patch and I have to confess I
just cannot.
As far as I can read, the point is that if we reach
SnapBuildProcessRunningXacts() when db_specific is true (which means
standby_decode is called in an output plugin that has set
need_shared_catalogs to false), _and_ we've not reached consistent state
yet, then we'll call LogStandbySnapshot with our DB oid to emit a new
xl_running_xacts message.
So the WAL-decoding process emits WAL. I don't know if in normal
conditions logical decoding processes emit WAL. If this is exceptional,
I think we should add a comment.
Now, this additional WAL message will be processed by all other
processes decoding WAL. Perhaps it will ignored by most of them. But
most importantly, it will also reach back to ourselves, at which point
we can hopefully use it to see that we might have reached consistent
state within our database. Then we know our snapshot is ready to be
used.
Is this correct?
I think the reason it's safe to skip a lot of the processing caused by
this additional process, is that xl_running_xacts messages are also
emitted in other places in a non-database specific manner. So all the
other placecs that are emitting that message continue to exist and
cause logical-decoders operate in the same way as before.
I think we should sprinkle lots of comments in several places about
this. For example, I propose that standby_redo() should have something
like
* If 'dbid' is valid, only gather transactions running in that database.
+ * Such records should not be the only ones emitted, because this has
+ * potentially dangerous side-effects which makes some places ignore them:
+ *
+ * 1. SnapBuildProcessRunningXacts will skip computing the xmin and restart
+ * point from its input record if the record's xmin is older that the
+ * snapbuilder's current xmin; this should normally be fine because that
+ * information will be updated from other xl_running_xacts records.
+ * 2. standby_redo will likewise skip processing such a record
*
(are there other things that should be mentioned?)
Also, LogStandbySnapshot() should have a comment explaining that passing
a valid dboid is a weird corner case which is to be used with care, and
that functions X Y and Z are going to ignore snapshots carrying a valid
dbid.
Why do we call SnapBuildFindSnapshot() to do this, instead of doing it
directly in SnapBuildProcessRunningXacts? Seems like it would be more
straightforward.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-05 20:41 ` Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-05 20:41 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
So here's a v55 version of the base REPACK patches that I'm feeling
comfortable calling very close to committable. I'm going to give an
additional read tomorrow and maybe make cosmetic adjustments, but there
should be nothing substantial. Of course, the subsequent additions in
the other patches of v54 are still in the cards, and they are most
likely essential.
Changes compared to v54:
- changed reform_tuple() to not deform the tuple if no attributes are
going to be touched. We can simply make a copy instead, which I
suspect is considerably cheaper (but I didn't measure).
- cleaned up worker shmem shutdown callback. I think it's how it is
because it copied parallel worker code, but that has a weird structure
for --as far as I can see-- no good reason (we oughta change it too)
- renamed the worker from pgoutput_repack to pgrepack.
(Note that this is an internal name that users don't face.)
- reverted some unnecessary changes to master
Thanks
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 09:15 ` vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: vignesh C @ 2026-04-06 09:15 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Mon, 6 Apr 2026 at 02:12, Alvaro Herrera <[email protected]> wrote:
>
> Hi,
>
> So here's a v55 version of the base REPACK patches that I'm feeling
> comfortable calling very close to committable. I'm going to give an
> additional read tomorrow and maybe make cosmetic adjustments, but there
> should be nothing substantial. Of course, the subsequent additions in
> the other patches of v54 are still in the cards, and they are most
> likely essential.
Few comments:
1) Can we add a comment why we should error out here, as repack
concurrently requires this whereas repack does not require this check.
Even if it is required for decoding can't it be handled by replica
identity full:
+ /*
+ * If the identity index is not set due to replica identity being, PK
+ * might exist.
+ */
+ ident_idx = RelationGetReplicaIndex(rel);
+ if (!OidIsValid(ident_idx) && OidIsValid(rel->rd_pkindex))
+ ident_idx = rel->rd_pkindex;
+ if (!OidIsValid(ident_idx))
+ ereport(ERROR,
+
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot process relation \"%s\"",
+ RelationGetRelationName(rel)),
+ errhint("Relation \"%s\" has no
identity index.",
+ RelationGetRelationName(rel)));
2) Do you think it will be good to add a test to simulate a case where
one of the swap_replation_files is successful and a failure after
that. We can verify that the oid should still point to old oids:
+ /*
+ * Even ShareUpdateExclusiveLock should have prevented others from
+ * creating / dropping indexes (even using the CONCURRENTLY
option), so we
+ * do not need to check whether the lists match.
+ */
+ forboth(lc, ind_oids_old, lc2, ind_oids_new)
+ {
+ Oid ind_old = lfirst_oid(lc);
+ Oid ind_new = lfirst_oid(lc2);
+ Oid mapped_tables[4] = {0};
+
+ swap_relation_files(ind_old, ind_new,
+ (old_table_oid
== RelationRelationId),
+ false, /*
swap_toast_by_content */
+ true,
+ InvalidTransactionId,
+ InvalidMultiXactId,
+ mapped_tables);
3) I'm not sure if this change should be part of this patch:
diff --git a/src/backend/storage/lmgr/generate-lwlocknames.pl
b/src/backend/storage/lmgr/generate-lwlocknames.pl
index b49007167b0..2e7f1054e62 100644
--- a/src/backend/storage/lmgr/generate-lwlocknames.pl
+++ b/src/backend/storage/lmgr/generate-lwlocknames.pl
@@ -162,7 +162,7 @@ while (<$lwlocklist>)
die
"$wait_event_lwlocks[$lwlock_count] defined in wait_event_names.txt but "
- . " missing from lwlocklist.h"
+ . "missing from lwlocklist.h"
if $lwlock_count < scalar @wait_event_lwlocks;
4) Can we add an example for concurrently in documentation
5) Typos
5.a) "jsut" should be "just":
+ * operation to avoid any lock-upgrade hazards. In the concurrent case, we
+ * grab ShareUpdateExclusiveLock (jsut like VACUUM) for most of the
5.b In commit message "intial" should be "initial"
While the "concurrent data" changes are applied at specific stages (we cannot
do that until the intial copy is finished and indexes are built), a background
6) This includes are not required in repack.c, for me it could compile
without it:
+#include "access/detoast.h"
+#include "access/xloginsert.h"
+#include "catalog/pg_control.h"
7) Can you check if the copyright year mentioned for the new files are
correct, as different files mention different years like:
/*-------------------------------------------------------------------------
*
* pgrepack.c
* Logical Replication output plugin for REPACK command
*
* Copyright (c) 2012-2026, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/pgrepack/pgrepack.c
*
*-------------------------------------------------------------------------
*/
/*-------------------------------------------------------------------------
*
* repack_worker.c
* Implementation of the background worker for ad-hoc logical decoding
* during REPACK (CONCURRENTLY).
*
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994-5, Regents of the University of California
*
*
* IDENTIFICATION
* src/backend/commands/repack_worker.c
Meson.build file:
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
Regards,
Vignesh
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
@ 2026-04-06 09:38 ` Alvaro Herrera <[email protected]>
2026-04-06 10:01 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 11:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-06 11:56 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 10:20 ` Re: Adding REPACK [concurrently] Tomas Vondra <[email protected]>
0 siblings, 5 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-06 09:38 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-06, vignesh C wrote:
> On Mon, 6 Apr 2026 at 02:12, Alvaro Herrera <[email protected]> wrote:
> Few comments:
Thanks for reviewing this patch!
> 1) Can we add a comment why we should error out here, as repack
> concurrently requires this whereas repack does not require this check.
> Even if it is required for decoding can't it be handled by replica
> identity full:
> + /*
> + * If the identity index is not set due to replica identity being, PK
> + * might exist.
> + */
> + ident_idx = RelationGetReplicaIndex(rel);
> + if (!OidIsValid(ident_idx) && OidIsValid(rel->rd_pkindex))
> + ident_idx = rel->rd_pkindex;
> + if (!OidIsValid(ident_idx))
> + ereport(ERROR,
> +
> errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("cannot process relation \"%s\"",
> + RelationGetRelationName(rel)),
> + errhint("Relation \"%s\" has no
> identity index.",
> + RelationGetRelationName(rel)));
Ah, I was just rewriting that comment moments ago. I think we could
make it work with replica identity full in theory, but I'm guessing it
would be useless. You really need an index that lets you locate the
affected tuples; otherwise the replay would take forever. If the table
is big, that would make the whole thing unworkable. If the table isn't
big, then you can probably just use straight repack without too much
disruption.
/*
* Obtain the replica identity index -- either one that has been set
* explicitly, or the primary key. If none of these cases apply, the
* table cannot be repacked concurrently. It might be possible to have
* repack work with a FULL replica identity; however that requires more
* work and is not implemented yet.
*/
Now, it's possible to have a non-unique index on some columns (not good
enough to be replica identity) which gives you a list of candidate
tuples, and then the implementation chooses one based on the other
columns which are present in the FULL replica identity. (This seems a
bit dangerous, because there might be multiple matching tuples; and how
do you choose?) Now let's further suppose you can narrow down to a
single tuple; in that case, using FULL would work. However, as I said,
this requires more implementation effort. We could entertain a patch
for this during the pg20 cycle, though I'm doubtful that it would really
be worth your while.
> 2) Do you think it will be good to add a test to simulate a case where
> one of the swap_replation_files is successful and a failure after
> that. We can verify that the oid should still point to old oids:
Hmm, it's not clear to me in which cases this can happen. Are you
thinking that the first swap_replation_files call dies because of
out-of-memory?
Note that the really weird cases, like pg_class or mapped relations, are
directly rejected. So we don't get into the branch with
!RelFileNumberIsValid, and so on.
I mean -- I'm not opposed to adding a test case for it. But I suspect
it's going to be somewhat annoying to write.
> 3) I'm not sure if this change should be part of this patch:
> diff --git a/src/backend/storage/lmgr/generate-lwlocknames.pl
> b/src/backend/storage/lmgr/generate-lwlocknames.pl
> index b49007167b0..2e7f1054e62 100644
> --- a/src/backend/storage/lmgr/generate-lwlocknames.pl
> +++ b/src/backend/storage/lmgr/generate-lwlocknames.pl
> @@ -162,7 +162,7 @@ while (<$lwlocklist>)
>
> die
> "$wait_event_lwlocks[$lwlock_count] defined in wait_event_names.txt but "
> - . " missing from lwlocklist.h"
> + . "missing from lwlocklist.h"
> if $lwlock_count < scalar @wait_event_lwlocks;
Yeah, will remove.
> 4) Can we add an example for concurrently in documentation
> 5) Typos
Sure.
> 6) This includes are not required in repack.c, for me it could compile
> without it:
> +#include "access/detoast.h"
> +#include "access/xloginsert.h"
> +#include "catalog/pg_control.h"
> 7) Can you check if the copyright year mentioned for the new files are
> correct, as different files mention different years like:
I'll look into these, thanks for pointing it out.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Linux transformó mi computadora, de una `máquina para hacer cosas',
en un aparato realmente entretenido, sobre el cual cada día aprendo
algo nuevo" (Jaime Salinas)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 10:01 ` vignesh C <[email protected]>
4 siblings, 0 replies; 156+ messages in thread
From: vignesh C @ 2026-04-06 10:01 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Mon, 6 Apr 2026 at 15:08, Alvaro Herrera <[email protected]> wrote:
>
> On 2026-Apr-06, vignesh C wrote:
>
> > On Mon, 6 Apr 2026 at 02:12, Alvaro Herrera <[email protected]> wrote:
>
> > Few comments:
>
> Thanks for reviewing this patch!
>
> > 1) Can we add a comment why we should error out here, as repack
> > concurrently requires this whereas repack does not require this check.
> > Even if it is required for decoding can't it be handled by replica
> > identity full:
> > + /*
> > + * If the identity index is not set due to replica identity being, PK
> > + * might exist.
> > + */
> > + ident_idx = RelationGetReplicaIndex(rel);
> > + if (!OidIsValid(ident_idx) && OidIsValid(rel->rd_pkindex))
> > + ident_idx = rel->rd_pkindex;
> > + if (!OidIsValid(ident_idx))
> > + ereport(ERROR,
> > +
> > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> > + errmsg("cannot process relation \"%s\"",
> > + RelationGetRelationName(rel)),
> > + errhint("Relation \"%s\" has no
> > identity index.",
> > + RelationGetRelationName(rel)));
>
> Ah, I was just rewriting that comment moments ago. I think we could
> make it work with replica identity full in theory, but I'm guessing it
> would be useless. You really need an index that lets you locate the
> affected tuples; otherwise the replay would take forever. If the table
> is big, that would make the whole thing unworkable. If the table isn't
> big, then you can probably just use straight repack without too much
> disruption.
>
> /*
> * Obtain the replica identity index -- either one that has been set
> * explicitly, or the primary key. If none of these cases apply, the
> * table cannot be repacked concurrently. It might be possible to have
> * repack work with a FULL replica identity; however that requires more
> * work and is not implemented yet.
> */
Should this be mentioned in XXX comment:
It might be possible to have repack work with a FULL replica identity;
however that requires more work and is not implemented yet.
> > 2) Do you think it will be good to add a test to simulate a case where
> > one of the swap_replation_files is successful and a failure after
> > that. We can verify that the oid should still point to old oids:
>
> Hmm, it's not clear to me in which cases this can happen. Are you
> thinking that the first swap_replation_files call dies because of
> out-of-memory?
Yes, I was thinking of that.
> Note that the really weird cases, like pg_class or mapped relations, are
> directly rejected. So we don't get into the branch with
> !RelFileNumberIsValid, and so on.
>
> I mean -- I'm not opposed to adding a test case for it. But I suspect
> it's going to be somewhat annoying to write.
I will verify this scenario through debugger.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 11:23 ` Antonin Houska <[email protected]>
4 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-06 11:23 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-06, vignesh C wrote:
> > 2) Do you think it will be good to add a test to simulate a case where
> > one of the swap_replation_files is successful and a failure after
> > that. We can verify that the oid should still point to old oids:
>
> Hmm, it's not clear to me in which cases this can happen. Are you
> thinking that the first swap_replation_files call dies because of
> out-of-memory?
>
> Note that the really weird cases, like pg_class or mapped relations, are
> directly rejected. So we don't get into the branch with
> !RelFileNumberIsValid, and so on.
>
> I mean -- I'm not opposed to adding a test case for it. But I suspect
> it's going to be somewhat annoying to write.
After all, I think we'd end up testing whether transaction abort works
correctly.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 11:56 ` Amit Kapila <[email protected]>
4 siblings, 0 replies; 156+ messages in thread
From: Amit Kapila @ 2026-04-06 11:56 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Mon, Apr 6, 2026 at 3:08 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2026-Apr-06, vignesh C wrote:
>
> > On Mon, 6 Apr 2026 at 02:12, Alvaro Herrera <[email protected]> wrote:
>
> > Few comments:
>
> Thanks for reviewing this patch!
>
> > 1) Can we add a comment why we should error out here, as repack
> > concurrently requires this whereas repack does not require this check.
> > Even if it is required for decoding can't it be handled by replica
> > identity full:
> > + /*
> > + * If the identity index is not set due to replica identity being, PK
> > + * might exist.
> > + */
> > + ident_idx = RelationGetReplicaIndex(rel);
> > + if (!OidIsValid(ident_idx) && OidIsValid(rel->rd_pkindex))
> > + ident_idx = rel->rd_pkindex;
> > + if (!OidIsValid(ident_idx))
> > + ereport(ERROR,
> > +
> > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> > + errmsg("cannot process relation \"%s\"",
> > + RelationGetRelationName(rel)),
> > + errhint("Relation \"%s\" has no
> > identity index.",
> > + RelationGetRelationName(rel)));
>
> Ah, I was just rewriting that comment moments ago. I think we could
> make it work with replica identity full in theory, but I'm guessing it
> would be useless. You really need an index that lets you locate the
> affected tuples; otherwise the replay would take forever. If the table
> is big, that would make the whole thing unworkable. If the table isn't
> big, then you can probably just use straight repack without too much
> disruption.
>
> /*
> * Obtain the replica identity index -- either one that has been set
> * explicitly, or the primary key. If none of these cases apply, the
> * table cannot be repacked concurrently. It might be possible to have
> * repack work with a FULL replica identity; however that requires more
> * work and is not implemented yet.
> */
>
> Now, it's possible to have a non-unique index on some columns (not good
> enough to be replica identity) which gives you a list of candidate
> tuples, and then the implementation chooses one based on the other
> columns which are present in the FULL replica identity. (This seems a
> bit dangerous, because there might be multiple matching tuples; and how
> do you choose?) Now let's further suppose you can narrow down to a
> single tuple; in that case, using FULL would work.
>
I think this is already working for apply worker where we compare
tuples if the index is non_uniuqe, see FindReplTupleInLocalRel.
>
However, as I said,
> this requires more implementation effort. We could entertain a patch
> for this during the pg20 cycle, though I'm doubtful that it would really
> be worth your while.
>
It is fine to support it later in pg20. In general, I wonder if we
have evaluated whether some of the apply-worker infrastructure could
have been reused here?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 21:11 ` Andres Freund <[email protected]>
2026-04-06 21:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
4 siblings, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-06 21:11 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Noah Misch <[email protected]>; +Cc: vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
I just saw this got committed and wanted to briefly play with it. It works
nicely!
Except that at first I tried this in a debugging build, and was briefly rather
dismayed by the performance. It was really slow. But it's not really related
to repack / the patches here.
Turns out that it is to a good part due to
heap_insert()
->CacheInvalidateHeapTuple()
->CacheInvalidateHeapTupleCommon()
->AssertCouldGetRelation()
not being cheap and running a *lot*.
Admittedly it's way worse if you build with -O0, which I tend to do to make
debugging easier.
In that config, the assert single-handled increases the time for a repack by
35% or so.
Noah, is there any reason we need to do the AssertCouldGetRelation() before
the !IsCatalogRelation(relation)? Given that the goal is to make
RelationGetRelid() safe, it doesn't seem there is?
It's totally valid to not have done so initially, this is a quite complicated
feature:
I saw this is using individual heap_insert()s during the
heapam_relation_copy_for_cluster(). Doing individual WAL logged inserts isn't
exactly cheap or efficient from a WAL volume perspective...
Is there anything other than round tuits preventing us from using
multi_insert?
That actually would also reduce the cost in the REPACK decoding worker, due to
having to parse far fewer WAL records.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-06 21:59 ` Alvaro Herrera <[email protected]>
2026-04-07 09:39 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-06 21:59 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Noah Misch <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-06, Andres Freund wrote:
> I just saw this got committed and wanted to briefly play with it. It works
> nicely!
Yeah, I have to say that Antonin did a great job here.
> Except that at first I tried this in a debugging build, and was briefly rather
> dismayed by the performance. It was really slow. But it's not really related
> to repack / the patches here.
>
> In that config, the assert single-handled increases the time for a repack by
> 35% or so.
Yeah, I saw it was kinda sluggish, but wow, I didn't see *that* much
overhead.
> It's totally valid to not have done so initially, this is a quite complicated
> feature:
>
> I saw this is using individual heap_insert()s during the
> heapam_relation_copy_for_cluster(). Doing individual WAL logged inserts isn't
> exactly cheap or efficient from a WAL volume perspective...
>
> Is there anything other than round tuits preventing us from using
> multi_insert?
>
> That actually would also reduce the cost in the REPACK decoding worker, due to
> having to parse far fewer WAL records.
Nope, not really ... but I don't have any :-(
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-06 21:59 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 09:39 ` Antonin Houska <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 09:39 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-06, Andres Freund wrote:
>
> > I just saw this got committed and wanted to briefly play with it. It works
> > nicely!
>
> Yeah, I have to say that Antonin did a great job here.
Likewise, I appreciate all your improvements! And of course, all the reviews
and testing done by other developers.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-07 01:10 ` Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Noah Misch @ 2026-04-07 01:10 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Mon, Apr 06, 2026 at 05:11:30PM -0400, Andres Freund wrote:
> heap_insert()
> ->CacheInvalidateHeapTuple()
> ->CacheInvalidateHeapTupleCommon()
> ->AssertCouldGetRelation()
> not being cheap and running a *lot*.
>
> Admittedly it's way worse if you build with -O0, which I tend to do to make
> debugging easier.
>
> In that config, the assert single-handled increases the time for a repack by
> 35% or so.
>
>
> Noah, is there any reason we need to do the AssertCouldGetRelation() before
> the !IsCatalogRelation(relation)? Given that the goal is to make
> RelationGetRelid() safe, it doesn't seem there is?
By running AssertCouldGetRelation() during every INSERT statement, this
detects cases that would be unsafe when the target of the INSERT happens to be
a system catalog. Little of our INSERT/UPDATE coverage targets a system
catalog. Hence, the current position is better for detection.
I wonder if this got slower in v19. In v14-v18, the assert's cost is
proportional to the number of held lwlocks, often 0 or 1. In v19, it's
proportional to PrivateRefCountHash cardinality.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
@ 2026-04-07 01:58 ` Andres Freund <[email protected]>
2026-04-07 02:42 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-07 01:58 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-06 18:10:56 -0700, Noah Misch wrote:
> On Mon, Apr 06, 2026 at 05:11:30PM -0400, Andres Freund wrote:
> > heap_insert()
> > ->CacheInvalidateHeapTuple()
> > ->CacheInvalidateHeapTupleCommon()
> > ->AssertCouldGetRelation()
> > not being cheap and running a *lot*.
> >
> > Admittedly it's way worse if you build with -O0, which I tend to do to make
> > debugging easier.
> >
> > In that config, the assert single-handled increases the time for a repack by
> > 35% or so.
> >
> >
> > Noah, is there any reason we need to do the AssertCouldGetRelation() before
> > the !IsCatalogRelation(relation)? Given that the goal is to make
> > RelationGetRelid() safe, it doesn't seem there is?
>
> By running AssertCouldGetRelation() during every INSERT statement, this
> detects cases that would be unsafe when the target of the INSERT happens to be
> a system catalog.
I see.
> Little of our INSERT/UPDATE coverage targets a system catalog.
Sure. We do have plenty DML doing heap_insert/update however.
> Hence, the current position is better for detection.
What if we returned early in AssertBufferLocksPermitCatalogRead() if
InterruptHoldoffCount == 0? That'd only fail if some code manually did a
RESUME_INTERRUPTS() to balance the one acquired as part of the content lock?
> I wonder if this got slower in v19. In v14-v18, the assert's cost is
> proportional to the number of held lwlocks, often 0 or 1. In v19, it's
> proportional to PrivateRefCountHash cardinality.
Yea, plausible. It will only scan PrivateRefCountHash if
PrivateRefCountOverflowed overflowed, but it did overflow in the case I was
testing...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-07 02:42 ` Noah Misch <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Noah Misch @ 2026-04-07 02:42 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Mon, Apr 06, 2026 at 09:58:19PM -0400, Andres Freund wrote:
> On 2026-04-06 18:10:56 -0700, Noah Misch wrote:
> > Hence, the current position is better for detection.
>
> What if we returned early in AssertBufferLocksPermitCatalogRead() if
> InterruptHoldoffCount == 0? That'd only fail if some code manually did a
> RESUME_INTERRUPTS() to balance the one acquired as part of the content lock?
Sounds good.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-07 04:44 ` Tom Lane <[email protected]>
2026-04-07 08:40 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Tom Lane @ 2026-04-07 04:44 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Noah Misch <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Maybe you saw this already, but BF member skink is failing on
src/test/modules/injection_points/specs/repack.spec:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2026-04-06%2022%3A50%3A41
regards, tom lane
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
@ 2026-04-07 08:40 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 08:40 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-07, Tom Lane wrote:
> Maybe you saw this already, but BF member skink is failing on
> src/test/modules/injection_points/specs/repack.spec:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2026-04-06%2022%3A50%3A41
Thanks, I hadn't, and I cannot reproduce it locally. Anyway, the
valgrind report is
==1617044== VALGRINDERROR-BEGIN
==1617044== Syscall param pwrite64(buf) points to uninitialised byte(s)
==1617044== at 0x6704003: pwrite (pwrite64.c:25)
==1617044== by 0x44AAC72: pg_pwritev (pg_iovec.h:101)
==1617044== by 0x44AC6B5: FileWriteV (fd.c:2280)
==1617044== by 0x44A8EC4: FileWrite (fd.h:245)
==1617044== by 0x44A8EC4: BufFileDumpBuffer (buffile.c:538)
==1617044== by 0x44A9034: BufFileFlush (buffile.c:724)
==1617044== by 0x44A9661: BufFileClose (buffile.c:418)
==1617044== by 0x426C4DE: export_initial_snapshot (repack_worker.c:346)
==1617044== by 0x426CA17: RepackWorkerMain (repack_worker.c:145)
==1617044== by 0x441D3AD: BackgroundWorkerMain (bgworker.c:865)
==1617044== by 0x44219D9: postmaster_child_launch (launch_backend.c:265)
==1617044== Address 0x12d745e2 is 106 bytes inside a block of size 8,272 client-defined
==1617044== at 0x4661CFE: palloc (mcxt.c:1411)
==1617044== by 0x44A8C54: makeBufFileCommon (buffile.c:121)
==1617044== by 0x44A933F: BufFileCreateFileSet (buffile.c:272)
==1617044== by 0x426C4A5: export_initial_snapshot (repack_worker.c:341)
==1617044== by 0x426CA17: RepackWorkerMain (repack_worker.c:145)
==1617044== by 0x441D3AD: BackgroundWorkerMain (bgworker.c:865)
==1617044== by 0x44219D9: postmaster_child_launch (launch_backend.c:265)
==1617044== by 0x4423B6D: StartBackgroundWorker (postmaster.c:4197)
==1617044== by 0x4423DB8: maybe_start_bgworkers (postmaster.c:4362)
==1617044== by 0x4425119: LaunchMissingBackgroundProcesses (postmaster.c:3437)
==1617044== by 0x4426A75: ServerLoop (postmaster.c:1737)
==1617044== by 0x44280DC: PostmasterMain (postmaster.c:1412)
==1617044== Uninitialised value was created by a stack allocation
==1617044== at 0x4674D39: SerializeSnapshot (snapmgr.c:1737)
==1617044==
==1617044== VALGRINDERROR-END
and obviously BufFileCreateFileSet() is being called by existing code
already and it doesn't fail, so I *think* the problem might be that
SerializedSnapshotData has some padding bytes that are being written.
Maybe using palloc0() is enough? I'll try that.
If that doesn't silence skink, I guess my next step is to reproduce
skink's environment more precisely ...
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
@ 2026-04-07 08:42 ` Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-04-07 08:42 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Noah Misch <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi Tom,
On Tue, Apr 7, 2026 at 10:14 AM Tom Lane <[email protected]> wrote:
> Maybe you saw this already, but BF member skink is failing on
> src/test/modules/injection_points/specs/repack.spec:
>
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2026-04-06%2022%3A50%3A41
>
i looked into this , it seems like valgrind catches the uninitialised
padding bytes, which
repack worker is writing using BufFileWrite, it seems this fix solved the
problem.
diff --git a/src/backend/utils/time/snapmgr.c
b/src/backend/utils/time/snapmgr.c
index 2e6197f5f35..f5682b87626 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -1739,6 +1739,8 @@ SerializeSnapshot(Snapshot snapshot, char
*start_address)
Assert(snapshot->subxcnt >= 0);
+ MemSet(&serialized_snapshot, 0, sizeof(SerializedSnapshotData));
+
/* Copy all required fields */
serialized_snapshot.xmin = snapshot->xmin;
serialized_snapshot.xmax = snapshot->xmax;
thoughts?
--
Thanks,
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
@ 2026-04-07 08:57 ` Antonin Houska <[email protected]>
2026-04-07 09:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 09:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 11:19 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 3 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 08:57 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Srinath Reddy Sadipiralla <[email protected]> wrote:
> Hi Tom,
>
> On Tue, Apr 7, 2026 at 10:14 AM Tom Lane <[email protected]> wrote:
>
> Maybe you saw this already, but BF member skink is failing on
> src/test/modules/injection_points/specs/repack.spec:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2026-04-06%2022%3A50%3A41
>
> i looked into this , it seems like valgrind catches the uninitialised padding bytes, which
> repack worker is writing using BufFileWrite, it seems this fix solved the problem.
>
> diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
> index 2e6197f5f35..f5682b87626 100644
> --- a/src/backend/utils/time/snapmgr.c
> +++ b/src/backend/utils/time/snapmgr.c
> @@ -1739,6 +1739,8 @@ SerializeSnapshot(Snapshot snapshot, char *start_address)
>
> Assert(snapshot->subxcnt >= 0);
>
> + MemSet(&serialized_snapshot, 0, sizeof(SerializedSnapshotData));
> +
> /* Copy all required fields */
> serialized_snapshot.xmin = snapshot->xmin;
> serialized_snapshot.xmax = snapshot->xmax;
>
> thoughts?
Could you reproduce the failure in your environment?
I haven't thought of this explanation because BufFileWrite() only copies the
data to a buffer in the BufFile structure and BufFileDumpBuffer() writes the
buffer. Maybe valgrind is able to track the copying?
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-07 09:22 ` Alvaro Herrera <[email protected]>
2026-04-07 09:35 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 10:08 ` Re: Adding REPACK [concurrently] John Naylor <[email protected]>
2 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 09:22 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-07, Antonin Houska wrote:
> > @@ -1739,6 +1739,8 @@ SerializeSnapshot(Snapshot snapshot, char *start_address)
> >
> > Assert(snapshot->subxcnt >= 0);
> >
> > + MemSet(&serialized_snapshot, 0, sizeof(SerializedSnapshotData));
> > +
> > /* Copy all required fields */
> > serialized_snapshot.xmin = snapshot->xmin;
> > serialized_snapshot.xmax = snapshot->xmax;
> >
> > thoughts?
>
> Could you reproduce the failure in your environment?
Yeah, he did.
> I haven't thought of this explanation because BufFileWrite() only copies the
> data to a buffer in the BufFile structure and BufFileDumpBuffer() writes the
> buffer. Maybe valgrind is able to track the copying?
Yeah, apparently it keeps track of tainted bytes somehow. Clever.
The change to palloc0() that I was proposing did not fix the problem,
because the stack allocated struct overwrote those zeroes with the
uninitialized padding bytes.
I ended up with an equivalent fix to Srinath's -- zero-initializing
the stack-allocated struct, so that the bytes that end up copied by
memcpy() are all defined. Srinath confirmed that in his environment the
valgrind failure goes away, so I think we're good.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No tengo por qué estar de acuerdo con lo que pienso"
(Carlos Caszeli)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 09:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 09:35 ` Antonin Houska <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 09:35 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-07, Antonin Houska wrote:
>
> > I haven't thought of this explanation because BufFileWrite() only copies the
> > data to a buffer in the BufFile structure and BufFileDumpBuffer() writes the
> > buffer. Maybe valgrind is able to track the copying?
>
> Yeah, apparently it keeps track of tainted bytes somehow. Clever.
>
> The change to palloc0() that I was proposing did not fix the problem,
> because the stack allocated struct overwrote those zeroes with the
> uninitialized padding bytes.
>
> I ended up with an equivalent fix to Srinath's -- zero-initializing
> the stack-allocated struct, so that the bytes that end up copied by
> memcpy() are all defined. Srinath confirmed that in his environment the
> valgrind failure goes away, so I think we're good.
Thanks!
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 09:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 10:08 ` John Naylor <[email protected]>
2026-04-07 10:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: John Naylor @ 2026-04-07 10:08 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
mamba is failing with
repack.c: In function 'initialize_change_context':
repack.c:2946:7: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
2946 | (Datum) NULL);
| ^
cc1: all warnings being treated as errors
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 09:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 10:08 ` Re: Adding REPACK [concurrently] John Naylor <[email protected]>
@ 2026-04-07 10:30 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 10:30 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-Apr-07, John Naylor wrote:
> mamba is failing with
>
> repack.c: In function 'initialize_change_context':
> repack.c:2946:7: error: cast from pointer to integer of different size
> [-Werror=pointer-to-int-cast]
> 2946 | (Datum) NULL);
> | ^
> cc1: all warnings being treated as errors
Hmm, yeah that should have been "(Datum) 0". Pushed.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"The Postgresql hackers have what I call a "NASA space shot" mentality.
Quite refreshing in a world of "weekend drag racer" developers."
(Scott Marlowe)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-07 09:29 ` Antonin Houska <[email protected]>
2026-04-07 09:41 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 09:29 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Antonin Houska <[email protected]> wrote:
> Srinath Reddy Sadipiralla <[email protected]> wrote:
>
> > i looked into this , it seems like valgrind catches the uninitialised padding bytes, which
> > repack worker is writing using BufFileWrite, it seems this fix solved the problem.
> >
> > diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
> > index 2e6197f5f35..f5682b87626 100644
> > --- a/src/backend/utils/time/snapmgr.c
> > +++ b/src/backend/utils/time/snapmgr.c
> > @@ -1739,6 +1739,8 @@ SerializeSnapshot(Snapshot snapshot, char *start_address)
> >
> > Assert(snapshot->subxcnt >= 0);
> >
> > + MemSet(&serialized_snapshot, 0, sizeof(SerializedSnapshotData));
> > +
> > /* Copy all required fields */
> > serialized_snapshot.xmin = snapshot->xmin;
> > serialized_snapshot.xmax = snapshot->xmax;
> >
> > thoughts?
>
> Could you reproduce the failure in your environment?
>
> I haven't thought of this explanation because BufFileWrite() only copies the
> data to a buffer in the BufFile structure and BufFileDumpBuffer() writes the
> buffer. Maybe valgrind is able to track the copying?
Given this message, you may be right:
==1617044== Address 0x12d745e2 is 106 bytes inside a block of size 8,272 client-defined
In my environment, the 'buffer' field starts at offset 80 into the BufFile
structure. We first write 8 bytes into it
BufFileWrite(file, &snap_size, sizeof(snap_size));
followed by the snapshot. Since sizeof(SerializedSnapshotData) is 24, the
offset 106 should be the padding following the 'takenDuringRecovery' field.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 09:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-07 09:41 ` Srinath Reddy Sadipiralla <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-04-07 09:41 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, Apr 7, 2026 at 2:59 PM Antonin Houska <[email protected]> wrote:
> Antonin Houska <[email protected]> wrote:
>
> > Srinath Reddy Sadipiralla <[email protected]> wrote:
> >
> > > i looked into this , it seems like valgrind catches the uninitialised
> padding bytes, which
> > > repack worker is writing using BufFileWrite, it seems this fix solved
> the problem.
> > >
> > > diff --git a/src/backend/utils/time/snapmgr.c
> b/src/backend/utils/time/snapmgr.c
> > > index 2e6197f5f35..f5682b87626 100644
> > > --- a/src/backend/utils/time/snapmgr.c
> > > +++ b/src/backend/utils/time/snapmgr.c
> > > @@ -1739,6 +1739,8 @@ SerializeSnapshot(Snapshot snapshot, char
> *start_address)
> > >
> > > Assert(snapshot->subxcnt >= 0);
> > >
> > > + MemSet(&serialized_snapshot, 0, sizeof(SerializedSnapshotData));
> > > +
> > > /* Copy all required fields */
> > > serialized_snapshot.xmin = snapshot->xmin;
> > > serialized_snapshot.xmax = snapshot->xmax;
> > >
> > > thoughts?
> >
> > Could you reproduce the failure in your environment?
> >
> > I haven't thought of this explanation because BufFileWrite() only copies
> the
> > data to a buffer in the BufFile structure and BufFileDumpBuffer() writes
> the
> > buffer. Maybe valgrind is able to track the copying?
>
> Given this message, you may be right:
>
> ==1617044== Address 0x12d745e2 is 106 bytes inside a block of size 8,272
> client-defined
>
> In my environment, the 'buffer' field starts at offset 80 into the BufFile
> structure. We first write 8 bytes into it
>
> BufFileWrite(file, &snap_size, sizeof(snap_size));
>
> followed by the snapshot. Since sizeof(SerializedSnapshotData) is 24, the
> offset 106 should be the padding following the 'takenDuringRecovery' field.
>
yeah , same in my environment aslo
==00:00:00:06.569 3479320== Syscall param pwrite64(buf) points to
uninitialised byte(s)
==00:00:00:06.569 3479320== at 0x55D6D38: pwrite (pwrite64.c:25)
==00:00:00:06.569 3479320== by 0x842EE7: pg_pwritev (pg_iovec.h:101)
==00:00:00:06.569 3479320== by 0x845F67: FileWriteV (fd.c:2280)
==00:00:00:06.569 3479320== by 0x840877: FileWrite (fd.h:245)
==00:00:00:06.569 3479320== by 0x841407: BufFileDumpBuffer
(buffile.c:538)
==00:00:00:06.569 3479320== by 0x841A67: BufFileFlush (buffile.c:724)
==00:00:00:06.569 3479320== by 0x8410B7: BufFileClose (buffile.c:418)
==00:00:00:06.569 3479320== by 0x4BFBBB: export_initial_snapshot
(repack_worker.c:346)
==00:00:00:06.569 3479320== by 0x4BF703: RepackWorkerMain
(repack_worker.c:145)
==00:00:00:06.569 3479320== by 0x765D3F: BackgroundWorkerMain
(bgworker.c:865)
==00:00:00:06.569 3479320== by 0x76C703: postmaster_child_launch
(launch_backend.c:265)
==00:00:00:06.569 3479320== by 0x774F87: StartBackgroundWorker
(postmaster.c:4197)
==00:00:00:06.569 3479320== Address 0x7b055f2 is 106 bytes inside a block
of size 8,272 client-defined
==00:00:00:06.569 3479320== at 0xB234F4: palloc (mcxt.c:1411)
==00:00:00:06.569 3479320== by 0x8408BF: makeBufFileCommon
(buffile.c:121)
==00:00:00:06.569 3479320== by 0x840C2F: BufFileCreateFileSet
(buffile.c:272)
==00:00:00:06.569 3479320== by 0x4BFB7F: export_initial_snapshot
(repack_worker.c:341)
==00:00:00:06.569 3479320== by 0x4BF703: RepackWorkerMain
(repack_worker.c:145)
==00:00:00:06.569 3479320== by 0x765D3F: BackgroundWorkerMain
(bgworker.c:865)
==00:00:00:06.569 3479320== by 0x76C703: postmaster_child_launch
(launch_backend.c:265)
==00:00:00:06.569 3479320== by 0x774F87: StartBackgroundWorker
(postmaster.c:4197)
==00:00:00:06.569 3479320== by 0x775277: maybe_start_bgworkers
(postmaster.c:4362)
==00:00:00:06.569 3479320== by 0x7739F7:
LaunchMissingBackgroundProcesses (postmaster.c:3437)
==00:00:00:06.569 3479320== by 0x770C2B: ServerLoop (postmaster.c:1737)
==00:00:00:06.569 3479320== by 0x77037B: PostmasterMain
(postmaster.c:1412)
==00:00:00:06.569 3479320== Uninitialised value was created by a stack
allocation
==00:00:00:06.569 3479320== at 0xB43008: SerializeSnapshot
(snapmgr.c:1737)
==00:00:00:06.569 3479320==
and you are spot on here with the explanation with offsets
Size snap_size; 8 bytes;
TransactionId xmin; 4 bytes
TransactionId xmax; 4 bytes
uint32 xcnt; 4 bytes
int32 subxcnt; 4 bytes
bool suboverflowed; 1 byte
bool takenDuringRecovery; 1 byte
/* 2 bytes of padding */
CommandId curcid; 4 bytes
--
Thanks,
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-07 11:19 ` Alvaro Herrera <[email protected]>
2026-04-07 12:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 11:19 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
There's one more failure in thorntail
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=thorntail&dt=2026-04-07%2006%3A31%3A03
because it runs with wal_level=minimal. I think we can make it work by
using a temp-config argument to run the tests, as in the attached.
I didn't actually try to run the buildfarm client though ...
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 21:11 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 01:10 ` Re: Adding REPACK [concurrently] Noah Misch <[email protected]>
2026-04-07 01:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 04:44 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-07 08:42 ` Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 11:19 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 12:17 ` Antonin Houska <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 12:17 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]>; vignesh C <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> There's one more failure in thorntail
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=thorntail&dt=2026-04-07%2006%3A31%3A03
> because it runs with wal_level=minimal. I think we can make it work by
> using a temp-config argument to run the tests, as in the attached.
LGTM. We had almost exactly this in earlier versions of the patch [1], before
commit 67c20979ce.
[1] https://www.postgresql.org/message-id/137668.1768235610%40localhost
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-08 10:20 ` Tomas Vondra <[email protected]>
2026-04-08 10:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
4 siblings, 1 reply; 156+ messages in thread
From: Tomas Vondra @ 2026-04-08 10:20 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
while building on a rpi5 with a 32-bit system, I'm getting these warnings:
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
In file included from ../../../src/include/access/tupmacs.h:20,
from ../../../src/include/access/htup_details.h:20,
from ../../../src/include/access/relscan.h:17,
from ../../../src/include/access/heapam.h:19,
from repack.c:36:
In function ‘VARSIZE_ANY’,
inlined from ‘restore_tuple’ at repack.c:2731:15:
../../../src/include/varatt.h:243:51: warning: array subscript
‘varattrib_4b[0]’ is partly outside array bounds of ‘union
<anonymous>[1]’ [-Warray-bounds=]
243 | ((((const varattrib_4b *) (PTR))->va_4byte.va_header >>
2) & 0x3FFFFFFF)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
../../../src/include/varatt.h:467:24: note: in expansion of macro
‘VARSIZE_4B’
467 | return VARSIZE_4B(PTR);
| ^~~~~~~~~~
repack.c: In function ‘restore_tuple’:
repack.c:2717:49: note: object ‘chunk_header’ of size 4
2717 | } chunk_header;
| ^~~~~~~~~~~~
In function ‘VARSIZE_ANY’,
inlined from ‘restore_tuple’ at repack.c:2734:4:
../../../src/include/varatt.h:243:51: warning: array subscript
‘varattrib_4b[0]’ is partly outside array bounds of ‘union
<anonymous>[1]’ [-Warray-bounds=]
243 | ((((const varattrib_4b *) (PTR))->va_4byte.va_header >>
2) & 0x3FFFFFFF)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
../../../src/include/varatt.h:467:24: note: in expansion of macro
‘VARSIZE_4B’
467 | return VARSIZE_4B(PTR);
| ^~~~~~~~~~
repack.c: In function ‘restore_tuple’:
repack.c:2717:49: note: object ‘chunk_header’ of size 4
2717 | } chunk_header;
| ^~~~~~~~~~~~
I'm not sure if it's just the compiler (gcc 14.2) being pesky, or if
it's an actual issue. The repack tests seem to pass fine.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-08 10:20 ` Re: Adding REPACK [concurrently] Tomas Vondra <[email protected]>
@ 2026-04-08 10:46 ` Antonin Houska <[email protected]>
2026-04-08 16:47 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-08 10:46 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Tomas Vondra <[email protected]> wrote:
> while building on a rpi5 with a 32-bit system, I'm getting these warnings:
>
> config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
> In file included from ../../../src/include/access/tupmacs.h:20,
> from ../../../src/include/access/htup_details.h:20,
> from ../../../src/include/access/relscan.h:17,
> from ../../../src/include/access/heapam.h:19,
> from repack.c:36:
> In function ‘VARSIZE_ANY’,
> inlined from ‘restore_tuple’ at repack.c:2731:15:
> ../../../src/include/varatt.h:243:51: warning: array subscript
> ‘varattrib_4b[0]’ is partly outside array bounds of ‘union
> <anonymous>[1]’ [-Warray-bounds=]
> 243 | ((((const varattrib_4b *) (PTR))->va_4byte.va_header >>
> 2) & 0x3FFFFFFF)
> | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
> ../../../src/include/varatt.h:467:24: note: in expansion of macro
> ‘VARSIZE_4B’
> 467 | return VARSIZE_4B(PTR);
> | ^~~~~~~~~~
> repack.c: In function ‘restore_tuple’:
> repack.c:2717:49: note: object ‘chunk_header’ of size 4
> 2717 | } chunk_header;
> | ^~~~~~~~~~~~
> In function ‘VARSIZE_ANY’,
> inlined from ‘restore_tuple’ at repack.c:2734:4:
> ../../../src/include/varatt.h:243:51: warning: array subscript
> ‘varattrib_4b[0]’ is partly outside array bounds of ‘union
> <anonymous>[1]’ [-Warray-bounds=]
> 243 | ((((const varattrib_4b *) (PTR))->va_4byte.va_header >>
> 2) & 0x3FFFFFFF)
> | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
> ../../../src/include/varatt.h:467:24: note: in expansion of macro
> ‘VARSIZE_4B’
> 467 | return VARSIZE_4B(PTR);
> | ^~~~~~~~~~
> repack.c: In function ‘restore_tuple’:
> repack.c:2717:49: note: object ‘chunk_header’ of size 4
> 2717 | } chunk_header;
> | ^~~~~~~~~~~~
> I'm not sure if it's just the compiler (gcc 14.2) being pesky, or if
> it's an actual issue. The repack tests seem to pass fine.
We already introduced this definition above in the function to suppress this
kind of warning
union
{
alignas(int32) varlena hdr;
char data[sizeof(void *)];
} chunk_header;
The problem on a 32-bit system probably is that sizeof(void *) is 4. We need
some other constant. Maybe (sizeof(varlena) + 1) ...
Thanks.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-08 10:20 ` Re: Adding REPACK [concurrently] Tomas Vondra <[email protected]>
2026-04-08 10:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-08 16:47 ` Tom Lane <[email protected]>
2026-04-09 06:59 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Tom Lane @ 2026-04-08 16:47 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Antonin Houska <[email protected]> writes:
> Tomas Vondra <[email protected]> wrote:
>> while building on a rpi5 with a 32-bit system, I'm getting these warnings:
>> ...
>> I'm not sure if it's just the compiler (gcc 14.2) being pesky, or if
>> it's an actual issue. The repack tests seem to pass fine.
Several 32-bit BF animals are showing these too (so far: dodo,
grison, and turaco).
> We already introduced this definition above in the function to suppress this
> kind of warning
> union
> {
> alignas(int32) varlena hdr;
> char data[sizeof(void *)];
> } chunk_header;
> The problem on a 32-bit system probably is that sizeof(void *) is 4. We need
> some other constant. Maybe (sizeof(varlena) + 1) ...
This seems unnecessarily Rube Goldberg-ish already. Why not just
uint64 chunk_header;
It will not hurt anything if the variable has more-than-required
alignment. And it'd be better if it were the same size everywhere.
regards, tom lane
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` Re: Adding REPACK [concurrently] vignesh C <[email protected]>
2026-04-06 09:38 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-08 10:20 ` Re: Adding REPACK [concurrently] Tomas Vondra <[email protected]>
2026-04-08 10:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-08 16:47 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
@ 2026-04-09 06:59 ` Antonin Houska <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-09 06:59 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Tom Lane <[email protected]> wrote:
> Antonin Houska <[email protected]> writes:
> > We already introduced this definition above in the function to suppress this
> > kind of warning
>
> > union
> > {
> > alignas(int32) varlena hdr;
> > char data[sizeof(void *)];
> > } chunk_header;
>
> > The problem on a 32-bit system probably is that sizeof(void *) is 4. We need
> > some other constant. Maybe (sizeof(varlena) + 1) ...
>
> This seems unnecessarily Rube Goldberg-ish already.
Indeed.
> Why not just
>
> uint64 chunk_header;
>
> It will not hurt anything if the variable has more-than-required
> alignment. And it'd be better if it were the same size everywhere.
That's certainly better. Thanks.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 10:14 ` Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-06 10:14 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Sun, Apr 5, 2026 at 10:41 PM Alvaro Herrera <[email protected]> wrote:
> So here's a v55 version of the base REPACK patches that I'm feeling
> comfortable calling very close to committable.
Some comments for v55.
repack.c:2725
if (!VARATT_IS_EXTERNAL(varlen))
continue;
I think it should be VARATT_IS_EXTERNAL_INDIRECT - the same as in pgrepack:244.
Also, after
natt_ext--;
I think it worth to add
Assert(natt_ext >= 0);
or Assert(natt_ext == 0); but in another place.
Or exit early with
for (int i = 0; i < desc->natts && natt_ext > 0; i++)
----------------------------
repack.c:2587
table_tuple_insert(rel, slot, GetCurrentCommandId(true),
HEAP_INSERT_NO_LOGICAL, NULL);
More idiomatic to use TABLE_INSERT_NO_LOGICAL instead.
----------------------------
repack.c:2696
ExecForceStoreHeapTuple(tup, slot, false);
AFAIU there is a memory leak here. Memory allocated above (for tuple)
is not freed in any way, because shouldFree == false.
Also, ExecClearTuple (tts_virtual_clear for virtual tuples) requires
TTS_SHOULDFREE to be set to free anything.
-------------------------
grab ShareUpdateExclusiveLock (jsut like VACUUM
typo in "just"
--------------------------
"If the identity index is not set due to replica identity being, PK"
Missing "FULL" after "being"?
-------------------------
Commit message:
"intial copy" -> "initial copy"
"backed performing REPACK" -> "backend performing REPACK"
Best regards,
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-06 10:48 ` Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-06 10:48 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-06, Mihail Nikalayeu wrote:
> repack.c:2725
>
> if (!VARATT_IS_EXTERNAL(varlen))
> continue;
>
> I think it should be VARATT_IS_EXTERNAL_INDIRECT - the same as in pgrepack:244.
Right.
> Also, after
>
> natt_ext--;
>
> I think it worth to add
>
> Assert(natt_ext >= 0);
>
> or Assert(natt_ext == 0); but in another place.
>
> Or exit early with
> for (int i = 0; i < desc->natts && natt_ext > 0; i++)
Hmm, how about something like this?
natt_ext--;
if (natt_ext < 0)
ereport(ERROR,
errcode(ERRCODE_DATA_CORRUPTED),
errmsg("insufficient number of attributes stored separately"));
I'd like to give more details, such as the tuple's identity, but that
seems hard ...
> ----------------------------
> repack.c:2587
>
> table_tuple_insert(rel, slot, GetCurrentCommandId(true),
> HEAP_INSERT_NO_LOGICAL, NULL);
>
> More idiomatic to use TABLE_INSERT_NO_LOGICAL instead.
Ah right.
> ----------------------------
> repack.c:2696
>
> ExecForceStoreHeapTuple(tup, slot, false);
>
> AFAIU there is a memory leak here. Memory allocated above (for tuple)
> is not freed in any way, because shouldFree == false.
> Also, ExecClearTuple (tts_virtual_clear for virtual tuples) requires
> TTS_SHOULDFREE to be set to free anything.
Yeah but I don't want the virtual tuple to be materialized (which would
happen in tts_virtual_materialize if I set shouldFree=true). The memory
should be freed in
ResetPerTupleExprContext(chgcxt->cc_estate);
anyway, right? Maybe deserves a comment.
> -------------------------
> grab ShareUpdateExclusiveLock (jsut like VACUUM
>
> typo in "just"
Right.
> --------------------------
>
> "If the identity index is not set due to replica identity being, PK"
>
> Missing "FULL" after "being"?
Ah yeah, I rewrote this.
> -------------------------
>
> Commit message:
>
> "intial copy" -> "initial copy"
> "backed performing REPACK" -> "backend performing REPACK"
Thanks!
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Sallah, I said NO camels! That's FIVE camels; can't you count?"
(Indiana Jones)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 11:21 ` Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-06 11:21 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi!
On Mon, Apr 6, 2026 at 12:48 PM Alvaro Herrera <[email protected]> wrote:
> Hmm, how about something like this?
>
> natt_ext--;
> if (natt_ext < 0)
> ereport(ERROR,
> errcode(ERRCODE_DATA_CORRUPTED),
> errmsg("insufficient number of attributes stored separately"));
I think it is ok.
> Yeah but I don't want the virtual tuple to be materialized (which would
> happen in tts_virtual_materialize if I set shouldFree=true). The memory
> should be freed in
> ResetPerTupleExprContext(chgcxt->cc_estate);
> anyway, right? Maybe deserves a comment.
Not sure, ResetPerTupleExprContext resets just "ExecutorState".
But slots are created in another memory context.
Also, we can't reset slot->tts_mcxt itself - it will free the slot also.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-06 22:22 ` Alvaro Herrera <[email protected]>
2026-04-06 23:53 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:32 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-07 19:43 ` Re: Adding REPACK [concurrently] Robert Treat <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 5 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-06 22:22 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello,
On 2026-Apr-06, Mihail Nikalayeu wrote:
> > Yeah but I don't want the virtual tuple to be materialized (which would
> > happen in tts_virtual_materialize if I set shouldFree=true). The memory
> > should be freed in
> > ResetPerTupleExprContext(chgcxt->cc_estate);
> > anyway, right? Maybe deserves a comment.
>
> Not sure, ResetPerTupleExprContext resets just "ExecutorState".
> But slots are created in another memory context.
>
> Also, we can't reset slot->tts_mcxt itself - it will free the slot also.
So what I ended up doing, is to just not change to the slot's context in
restore_tuple. That was just dumb (mea culpa). So all the memory we
allocate for the slot lives in the executor context, and goes away on
the ResetPerTupleExprContext call at the bottom of the loop. BTW I
don't understand why you say that function only resets ExecutorState --
as far as I can tell, it does this
#define ResetPerTupleExprContext(estate) \
do { \
if ((estate)->es_per_tuple_exprcontext) \
ResetExprContext((estate)->es_per_tuple_exprcontext); \
} while (0)
which in turn does
#define ResetExprContext(econtext) \
MemoryContextReset((econtext)->ecxt_per_tuple_memory)
which AFAICT is exactly what we want.
Anyway, here's the three missing parts. I have not yet edited the
deadlock-checker one to protect autovacuum from processing tables under
repack.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 23:53 ` Mihail Nikalayeu <[email protected]>
4 siblings, 0 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-06 23:53 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi!
On Tue, Apr 7, 2026 at 12:22 AM Alvaro Herrera <[email protected]> wrote:
> So what I ended up doing, is to just not change to the slot's context in
> restore_tuple.
Yes, it should work.
> BTW I
> don't understand why you say that function only resets ExecutorState
My bad, I confused "ExprContext" with "ExecutorState" (names passed to
AllocSetContextCreate) while tracing.
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 12:15 ` Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
4 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-04-07 12:15 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, Apr 7, 2026 at 3:52 AM Alvaro Herrera <[email protected]> wrote:
>
> Anyway, here's the three missing parts. I have not yet edited the
> deadlock-checker one to protect autovacuum from processing tables under
> repack.
>
I have a question based on 0001's commit message: "This patch adds a
new option to logical replication output plugin, to declare that it
does not use shared catalogs (i.e. catalogs that can be changed by
transactions running in other databases in the cluster).". In which
cases, currently plugin needs to access multi-database transactions or
transactions that need to access shared catalogs and on what basis a
plugin can decide that the changes it requires won't need any such
access.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-07 12:33 ` Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-07 12:33 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-07, Amit Kapila wrote:
> I have a question based on 0001's commit message: "This patch adds a
> new option to logical replication output plugin, to declare that it
> does not use shared catalogs (i.e. catalogs that can be changed by
> transactions running in other databases in the cluster).". In which
> cases, currently plugin needs to access multi-database transactions or
> transactions that need to access shared catalogs and on what basis a
> plugin can decide that the changes it requires won't need any such
> access.
I don't think any plugin needs "multi-database" access as such, but
needing access to shared catalogs is likely normal. Repack knows it
won't access any shared catalogs, so it can set the flag at ease.
There's a cross-check added in the commit that tests for access to
shared catalogs if the flag is set to false. I guess you could set it
to false and see what breaks :-)
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 14:10 ` Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Andres Freund @ 2026-04-07 14:10 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-07 14:33:50 +0200, Alvaro Herrera wrote:
> On 2026-Apr-07, Amit Kapila wrote:
>
> > I have a question based on 0001's commit message: "This patch adds a
> > new option to logical replication output plugin, to declare that it
> > does not use shared catalogs (i.e. catalogs that can be changed by
> > transactions running in other databases in the cluster).". In which
> > cases, currently plugin needs to access multi-database transactions or
> > transactions that need to access shared catalogs and on what basis a
> > plugin can decide that the changes it requires won't need any such
> > access.
>
> I don't think any plugin needs "multi-database" access as such, but
> needing access to shared catalogs is likely normal. Repack knows it
> won't access any shared catalogs, so it can set the flag at ease.
>
> There's a cross-check added in the commit that tests for access to
> shared catalogs if the flag is set to false. I guess you could set it
> to false and see what breaks :-)
I think this has a quite high chance of indirect breakages. You just need some
cache invalidation processing / building accessing shared catalogs to violate
the rule, and whether that happens very heavily depends on what cache entries
are present and whether something registers relcache callbacks or such.
This can be triggered by an output function during logical decoding or such,
so you don't really have control over it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-07 15:38 ` Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 15:38 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Andres Freund <[email protected]> wrote:
> On 2026-04-07 14:33:50 +0200, Alvaro Herrera wrote:
> > On 2026-Apr-07, Amit Kapila wrote:
> >
> > > I have a question based on 0001's commit message: "This patch adds a
> > > new option to logical replication output plugin, to declare that it
> > > does not use shared catalogs (i.e. catalogs that can be changed by
> > > transactions running in other databases in the cluster).". In which
> > > cases, currently plugin needs to access multi-database transactions or
> > > transactions that need to access shared catalogs and on what basis a
> > > plugin can decide that the changes it requires won't need any such
> > > access.
> >
> > I don't think any plugin needs "multi-database" access as such, but
> > needing access to shared catalogs is likely normal. Repack knows it
> > won't access any shared catalogs, so it can set the flag at ease.
> >
> > There's a cross-check added in the commit that tests for access to
> > shared catalogs if the flag is set to false. I guess you could set it
> > to false and see what breaks :-)
>
> I think this has a quite high chance of indirect breakages. You just need some
> cache invalidation processing / building accessing shared catalogs to violate
> the rule, and whether that happens very heavily depends on what cache entries
> are present and whether something registers relcache callbacks or such.
>
> This can be triggered by an output function during logical decoding or such,
> so you don't really have control over it.
The REPACK plugin only deforms tuples and writes them to a file, so I think
that things like this should not happen. However, I admit that an option that
allows the plugin developer to declare "I don't need shared catalogs" may be
considered deceptive.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-07 15:48 ` Andres Freund <[email protected]>
2026-04-07 16:01 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-07 15:48 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-07 17:38:24 +0200, Antonin Houska wrote:
> Andres Freund <[email protected]> wrote:
>
> > On 2026-04-07 14:33:50 +0200, Alvaro Herrera wrote:
> > > On 2026-Apr-07, Amit Kapila wrote:
> > >
> > > > I have a question based on 0001's commit message: "This patch adds a
> > > > new option to logical replication output plugin, to declare that it
> > > > does not use shared catalogs (i.e. catalogs that can be changed by
> > > > transactions running in other databases in the cluster).". In which
> > > > cases, currently plugin needs to access multi-database transactions or
> > > > transactions that need to access shared catalogs and on what basis a
> > > > plugin can decide that the changes it requires won't need any such
> > > > access.
> > >
> > > I don't think any plugin needs "multi-database" access as such, but
> > > needing access to shared catalogs is likely normal. Repack knows it
> > > won't access any shared catalogs, so it can set the flag at ease.
> > >
> > > There's a cross-check added in the commit that tests for access to
> > > shared catalogs if the flag is set to false. I guess you could set it
> > > to false and see what breaks :-)
> >
> > I think this has a quite high chance of indirect breakages. You just need some
> > cache invalidation processing / building accessing shared catalogs to violate
> > the rule, and whether that happens very heavily depends on what cache entries
> > are present and whether something registers relcache callbacks or such.
> >
> > This can be triggered by an output function during logical decoding or such,
> > so you don't really have control over it.
>
> The REPACK plugin only deforms tuples and writes them to a file, so I think
> that things like this should not happen.
You don't need to do it yourself. It just requires a shared_preload_library
extension to register a relcache invalidation callback that accesses shared
catalog.
It's only kind of an accident that we don't have a case today that accesses
shared catcaches during a relcache build in core (I'm not even sure there's
nothing). You'd just need somebody to add e.g. relcache caching for
publications for that to change. Or look up information about a reloption in
pg_parameter_acl. Or lookup tablespace configuration.
> However, I admit that an option that allows the plugin developer to declare
> "I don't need shared catalogs" may be considered deceptive.
At the very least it would need to be a runtime check rather than just an
assert. This would much more likely to be hit in production because otherwise
it's probably hard to hit the case where shared invalidations happen in the
wrong moment. And the consequences are corrupted caches, which could cause
all kinds of havoc.
But I think this may need more infrastructure / deeper analysis than what we
can do right now.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-07 16:01 ` Antonin Houska <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 16:01 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Andres Freund <[email protected]> wrote:
> On 2026-04-07 17:38:24 +0200, Antonin Houska wrote:
> > The REPACK plugin only deforms tuples and writes them to a file, so I think
> > that things like this should not happen.
>
> You don't need to do it yourself. It just requires a shared_preload_library
> extension to register a relcache invalidation callback that accesses shared
> catalog.
>
> It's only kind of an accident that we don't have a case today that accesses
> shared catcaches during a relcache build in core (I'm not even sure there's
> nothing). You'd just need somebody to add e.g. relcache caching for
> publications for that to change. Or look up information about a reloption in
> pg_parameter_acl. Or lookup tablespace configuration.
>
>
> > However, I admit that an option that allows the plugin developer to declare
> > "I don't need shared catalogs" may be considered deceptive.
>
> At the very least it would need to be a runtime check rather than just an
> assert. This would much more likely to be hit in production because otherwise
> it's probably hard to hit the case where shared invalidations happen in the
> wrong moment. And the consequences are corrupted caches, which could cause
> all kinds of havoc.
>
>
> But I think this may need more infrastructure / deeper analysis than what we
> can do right now.
ok, thanks a lot for having looked at it.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-27 04:25 ` Amit Kapila <[email protected]>
2026-04-27 04:46 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Amit Kapila @ 2026-04-27 04:25 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, Apr 7, 2026 at 9:18 PM Andres Freund <[email protected]> wrote:
>
> On 2026-04-07 17:38:24 +0200, Antonin Houska wrote:
> > Andres Freund <[email protected]> wrote:
> >
> > > On 2026-04-07 14:33:50 +0200, Alvaro Herrera wrote:
> > > > On 2026-Apr-07, Amit Kapila wrote:
> > > >
> > > > > I have a question based on 0001's commit message: "This patch adds a
> > > > > new option to logical replication output plugin, to declare that it
> > > > > does not use shared catalogs (i.e. catalogs that can be changed by
> > > > > transactions running in other databases in the cluster).". In which
> > > > > cases, currently plugin needs to access multi-database transactions or
> > > > > transactions that need to access shared catalogs and on what basis a
> > > > > plugin can decide that the changes it requires won't need any such
> > > > > access.
> > > >
> > > > I don't think any plugin needs "multi-database" access as such, but
> > > > needing access to shared catalogs is likely normal. Repack knows it
> > > > won't access any shared catalogs, so it can set the flag at ease.
> > > >
> > > > There's a cross-check added in the commit that tests for access to
> > > > shared catalogs if the flag is set to false. I guess you could set it
> > > > to false and see what breaks :-)
> > >
> > > I think this has a quite high chance of indirect breakages. You just need some
> > > cache invalidation processing / building accessing shared catalogs to violate
> > > the rule, and whether that happens very heavily depends on what cache entries
> > > are present and whether something registers relcache callbacks or such.
> > >
> > > This can be triggered by an output function during logical decoding or such,
> > > so you don't really have control over it.
> >
> > The REPACK plugin only deforms tuples and writes them to a file, so I think
> > that things like this should not happen.
>
> You don't need to do it yourself. It just requires a shared_preload_library
> extension to register a relcache invalidation callback that accesses shared
> catalog.
>
> It's only kind of an accident that we don't have a case today that accesses
> shared catcaches during a relcache build in core (I'm not even sure there's
> nothing). You'd just need somebody to add e.g. relcache caching for
> publications for that to change. Or look up information about a reloption in
> pg_parameter_acl. Or lookup tablespace configuration.
>
>
> > However, I admit that an option that allows the plugin developer to declare
> > "I don't need shared catalogs" may be considered deceptive.
>
> At the very least it would need to be a runtime check rather than just an
> assert. This would much more likely to be hit in production because otherwise
> it's probably hard to hit the case where shared invalidations happen in the
> wrong moment. And the consequences are corrupted caches, which could cause
> all kinds of havoc.
>
>
> But I think this may need more infrastructure / deeper analysis than what we
> can do right now.
>
We still have not decided on this point. The code corresponding to
this part [1] has an open bug as well [2]. Based on the discussion
here, I don't see we have a consensus with the current approach and
even if we want to go with the current approach it seems that we need
more work which needs further analysis. Alvaro, others, what is your
take on this?
[1]: commit 0d3dba38c777384a9dd7dffe924355c9683a6b71: Allow logical
replication snapshots to be database-specific
[2]: https://www.postgresql.org/message-id/CAHg%2BQDcQak4jx_6X2_Ws98rzG%3DxBARLjqm_%3D56wTRUtNsY4DZQ%40ma...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-27 04:46 ` Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-04-27 04:46 UTC (permalink / raw)
To: 'Amit Kapila' <[email protected]>; Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Dear Hackers,
>
> We still have not decided on this point. The code corresponding to
> this part [1] has an open bug as well [2]. Based on the discussion
> here, I don't see we have a consensus with the current approach and
> even if we want to go with the current approach it seems that we need
> more work which needs further analysis. Alvaro, others, what is your
> take on this?
>
FYI, the point I raised [1] [2] seemed not to be fixed yet. Not sure it's the
last point we missed.
[1]: https://www.postgresql.org/message-id/225003.1775571560%40localhost
[2]: https://www.postgresql.org/message-id/CAA4eK1KWDbBk4FgbbWdivQLrPPzR4zgvfnHK4WjWC78rbuRVbg%40mail.gma...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-30 11:24 ` Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:04 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-30 11:24 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Andres Freund <[email protected]>; Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Mon, Apr 27, 2026 at 6:25 AM Amit Kapila <[email protected]> wrote:
> Alvaro, others, what is your take on this?
I agree with you here - we should AT LEAST make that an ERROR instead
of an assert and also check it during cache access (not only during
the scan because of cache misses).
But I think it will still be fragile in case of some extensions installed.
Anyway... We also have an issue with correctness right now.
I took the old stress test from [0] (the first two) and it fails now,
even with the fix from [1] ("Possible premature SNAPBUILD_CONSISTENT
with DB-specific running_xacts").
It looks like [1] fixes 008_repack_concurrently.pl, but
007_repack_concurrently.pl fails anyway, including
pgbench: error: client 1 script 0 aborted in command 10 query 0:
ERROR: could not create unique index "tbl_pkey_repacknew"
# DETAIL: Key (i)=(383) is duplicated.
and
'pgbench: error: pgbench:client 23 script 0 aborted in command 31
query 0: ERROR: division by zero
Last one is not MVCC-related; you can see from the logs that it
performs something like SELECT (509063) / 0 when the table sum
changes.
Setting need_shared_catalogs = true make them pass, so something is
wrong with its correctness.
P.S.
I think it is good idea to add these stress tests to the source tree,
perhaps with some kind PG_TEST_EXTRA=stress (as done in [1]).
[0]: https://www.postgresql.org/message-id/flat/CADzfLwUitd5J17O9FUxNGrZBurOpL6n%2BtnS6dgArXi-i9DNxhg%40m...
[1]: https://www.postgresql.org/message-id/flat/CAHg%2BQDcQak4jx_6X2_Ws98rzG%3DxBARLjqm_%3D56wTRUtNsY4DZQ...
[2]: https://www.postgresql.org/message-id/flat/CADzfLwWC%2BKxYWb-2QotWaz-q1LK8koLNVUR1Q8obAtt%2BR_sORA%4...
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-05-04 13:24 ` Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-05-04 13:24 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Mihail Nikalayeu <[email protected]> wrote:
> On Mon, Apr 27, 2026 at 6:25 AM Amit Kapila <[email protected]> wrote:
> > Alvaro, others, what is your take on this?
>
> I agree with you here - we should AT LEAST make that an ERROR instead
> of an assert and also check it during cache access (not only during
> the scan because of cache misses).
> But I think it will still be fragile in case of some extensions installed.
>
> Anyway... We also have an issue with correctness right now.
>
> I took the old stress test from [0] (the first two) and it fails now,
> even with the fix from [1] ("Possible premature SNAPBUILD_CONSISTENT
> with DB-specific running_xacts").
>
> It looks like [1] fixes 008_repack_concurrently.pl, but
> 007_repack_concurrently.pl fails anyway, including
>
> pgbench: error: client 1 script 0 aborted in command 10 query 0:
> ERROR: could not create unique index "tbl_pkey_repacknew"
> # DETAIL: Key (i)=(383) is duplicated.
> and
> 'pgbench: error: pgbench:client 23 script 0 aborted in command 31
> query 0: ERROR: division by zero
>
> Last one is not MVCC-related; you can see from the logs that it
> performs something like SELECT (509063) / 0 when the table sum
> changes.
>
> Setting need_shared_catalogs = true make them pass, so something is
> wrong with its correctness.
Thanks for testing again. Whether we keep the "database specific slots" or
not, it'd be good to know what exactly the reason of these errors is. I wonder
if the feature just exposes a problem that remains shadowed otherwise, due to
the contention on replication slot. I'm going to investigate.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-05 12:47 ` Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-05-05 12:47 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Antonin Houska <[email protected]> wrote:
> Mihail Nikalayeu <[email protected]> wrote:
>
> > On Mon, Apr 27, 2026 at 6:25 AM Amit Kapila <[email protected]> wrote:
> > > Alvaro, others, what is your take on this?
> >
> > I agree with you here - we should AT LEAST make that an ERROR instead
> > of an assert and also check it during cache access (not only during
> > the scan because of cache misses).
> > But I think it will still be fragile in case of some extensions installed.
> >
> > Anyway... We also have an issue with correctness right now.
> >
> > I took the old stress test from [0] (the first two) and it fails now,
> > even with the fix from [1] ("Possible premature SNAPBUILD_CONSISTENT
> > with DB-specific running_xacts").
> >
> > It looks like [1] fixes 008_repack_concurrently.pl, but
> > 007_repack_concurrently.pl fails anyway, including
> >
> > pgbench: error: client 1 script 0 aborted in command 10 query 0:
> > ERROR: could not create unique index "tbl_pkey_repacknew"
> > # DETAIL: Key (i)=(383) is duplicated.
> > and
> > 'pgbench: error: pgbench:client 23 script 0 aborted in command 31
> > query 0: ERROR: division by zero
> >
> > Last one is not MVCC-related; you can see from the logs that it
> > performs something like SELECT (509063) / 0 when the table sum
> > changes.
> >
> > Setting need_shared_catalogs = true make them pass, so something is
> > wrong with its correctness.
>
> Thanks for testing again. Whether we keep the "database specific slots" or
> not, it'd be good to know what exactly the reason of these errors is. I wonder
> if the feature just exposes a problem that remains shadowed otherwise, due to
> the contention on replication slot. I'm going to investigate.
I think the problem is that with database-specific snapshot,
SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
/*
* Database specific transaction info may exist to reach CONSISTENT state
* faster, however the code below makes no use of it. Moreover, such
* record might cause problems because the following normal (cluster-wide)
* record can have lower value of oldestRunningXid. In that case, let's
* wait with the cleanup for the next regular cluster-wide record.
*/
if (OidIsValid(running->dbid))
return;
and thus some transactions whose XID is below running->oldestRunningXid may
continue to be incorrectly considered running.
I originally thought that this should not happen because such transactions
will be added to the builder's array of committed transactions by
SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
a transaction listed in the xl_running_xacts WAL record is not guaranteed to
follow the xl_running_xacts record in WAL. In other words, even if
xl_running_xacts is created before a COMMIT record of the contained
transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
might not take place.
I've got no good idea how to fix that. Not sure I'm able to pursue the
"database-specific snapshots" feature now.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-05 15:02 ` Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-05 15:02 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-05, Antonin Houska wrote:
> However, I failed to notice that COMMIT record of
> a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> follow the xl_running_xacts record in WAL. In other words, even if
> xl_running_xacts is created before a COMMIT record of the contained
> transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> might not take place.
That's pretty bad news.
> I've got no good idea how to fix that. Not sure I'm able to pursue the
> "database-specific snapshots" feature now.
It appears that the only reasonable course of action at this point is to
revert 0d3dba38c777.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-06 08:25 ` Antonin Houska <[email protected]>
2026-05-08 12:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-05-06 08:25 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-May-05, Antonin Houska wrote:
>
> > However, I failed to notice that COMMIT record of
> > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > follow the xl_running_xacts record in WAL. In other words, even if
> > xl_running_xacts is created before a COMMIT record of the contained
> > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > might not take place.
>
> That's pretty bad news.
>
> > I've got no good idea how to fix that.
One idea occurred to me yet, effectively it's just a cleanup. Part of it was
already proposed [1].
[1] https://www.postgresql.org/message-id/flat/CAHg%2BQDcQak4jx_6X2_Ws98rzG%3DxBARLjqm_%3D56wTRUtNsY4DZQ...
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-08 12:25 ` Amit Kapila <[email protected]>
2026-05-08 13:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-08 12:25 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, May 6, 2026 at 1:55 PM Antonin Houska <[email protected]> wrote:
>
> Alvaro Herrera <[email protected]> wrote:
>
> > On 2026-May-05, Antonin Houska wrote:
> >
> > > However, I failed to notice that COMMIT record of
> > > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > > follow the xl_running_xacts record in WAL. In other words, even if
> > > xl_running_xacts is created before a COMMIT record of the contained
> > > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > > might not take place.
> >
> > That's pretty bad news.
> >
> > > I've got no good idea how to fix that.
>
> One idea occurred to me yet, effectively it's just a cleanup. Part of it was
> already proposed [1].
>
Some issues/inefficiencies regarding this fix and base code related to
db-specific snapshots built during decoding:
*
After fix, whenever a db-specific decoder sees a cluster-wide
xl_running_xacts record, it unconditionally calls
LogStandbySnapshot(MyDatabaseId) and returns. This triggers for every
cluster-wide record the decoder encounters (including post snapbuild's
CONSISTENT state) , for the entire duration of the decoding session.
LogStandbySnapshot acquires ProcArrayLock + XidGenLock, calls
GetRunningTransactionData, and writes WAL. With N active db-specific
decoding sessions, each cluster-wide record now triggers N additional
WAL writes.
Additionally, LogStandbySnapshot also logs AccessExclusiveLocks before
the running_xacts record. Physical standbys skip db-specific
XLOG_RUNNING_XACTS records via standby_redo(), but they do process the
preceding XLOG_STANDBY_LOCK records. The same locks may already have
been logged in the most recent cluster-wide snapshot. Physical
standbys could end up processing these lock records twice which may
not be harmful because I think we avoid re-acquiring the lock but
still it is a new overhead in the system.
*
When a cluster-wide running_xacts record arrives:
SnapBuildProcessRunningXacts calls LogStandbySnapshot and returns
early. ReorderBufferAbortOld is called, but with the cluster-wide
oldestRunningXid, which could lag far behind the db-specific value
(due to a long-running transaction in another database).
When a db-specific record arrives: SnapBuildProcessRunningXacts
processes it and advances builder->xmin with the db-specific (more
current) oldestRunningXid. But ReorderBufferAbortOld is NOT called for
db-specific records. This means the reorder buffer is cleaned up using
a conservative, potentially very old, cluster-wide oldestRunningXid,
even though builder->xmin has already advanced much further. The
reorder buffer holds stale entries longer than necessary, increasing
memory pressure.
*
I also see a design level problem with plugins that have
need_shared_catalogs=false and use failover slots. IIUC, the
db-specific optimization was designed around a live decoding session
on the primary which can emit and immediately read its own db-specific
records in the WAL stream to reach consistent state. The
LogicalSlotAdvanceAndCheckSnapState path used by slotsync has a
bounded WAL window and cannot exploit the feedback loop, making the
two mechanisms fundamentally incompatible. I know the slot created by
pgrepack doesn't enable failover option but we have not even added any
guards or thought about db-specific snapbuilds with other parts of the
system that rely on cluster-wide running_xact records, so there could
be more problems which we don't see with the current set of tests.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-08 12:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-08 13:58 ` Alvaro Herrera <[email protected]>
2026-05-08 23:22 ` Re: Adding REPACK [concurrently] Masahiko Sawada <[email protected]>
2026-05-10 11:24 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-08 13:58 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-08, Amit Kapila wrote:
> On Wed, May 6, 2026 at 1:55 PM Antonin Houska <[email protected]> wrote:
> > One idea occurred to me yet, effectively it's just a cleanup. Part of it was
> > already proposed [1].
Hmm, I think this cleanup makes sense. If I apply the test patches
(0001 and 0002 here), they fail almost immediately; but after applying
0003 all is again well. I think these tests are a good thing to have in
the tree, even if we end up reverting db-specific snapshots later.
After some back and forth, I modified the tests slightly so that
the search PG_TEST_EXTRA for the string "stress_concurrently=N". The N
can be changed so that the tests run for longer; if not given, it's
taken as 1, and the tests run for around 6 seconds (so N=10 means runs
for a minute). I think this is a convenient gadget for other tests of
this kind on CONCURRENTLY commands, such as the one proposed for CIC
elsewhere.
As written, these tests would run nowhere until we add that string in
some buildfarm animal. I debated with myself whether to assume N=1 when
the string is not given. I still think that's a good idea but perhaps
we should have something to prevent it from running by default when
under Valgrind or other slow things like that. In normal conditions,
the total runtime is not affected when they are run with N=1 as part of
the whole test suite.
> Some issues/inefficiencies regarding this fix and base code related to
> db-specific snapshots built during decoding: [...]
Thanks for spending time reviewing this code. I think none of these
problems are fundamental in nature, but they are obviously worth
addressing for 19. If we hit some roadblock, we can still revert only
db-specific snapshots.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-08 12:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-08 13:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-08 23:22 ` Masahiko Sawada <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Masahiko Sawada @ 2026-05-08 23:22 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Kapila <[email protected]>; Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Fri, May 8, 2026 at 6:58 AM Alvaro Herrera <[email protected]> wrote:
>
> On 2026-May-08, Amit Kapila wrote:
>
> > On Wed, May 6, 2026 at 1:55 PM Antonin Houska <[email protected]> wrote:
>
> > Some issues/inefficiencies regarding this fix and base code related to
> > db-specific snapshots built during decoding: [...]
>
> Thanks for spending time reviewing this code. I think none of these
> problems are fundamental in nature, but they are obviously worth
> addressing for 19. If we hit some roadblock, we can still revert only
> db-specific snapshots.
>
I'm still studying REPACK (CONCURRENTLY), and I have a question about
db-specific decoder; as far as I read the codes, any decoding plugin
can use db-specific decoder by setting need_shared_catalog=false but
should we prevent such slots from being created on standbys? I'm a bit
concerned that plugin developers inadvertently set
need_shared_catalog=false in the startup callback and users would
create such slots on standbys. For instance, if we create a logical
slot with pgrepack plugin on the standby, we would get an assertion
failure as the db-specific decoder tries to call LogStandbySnapshot()
via SnapBuildProcessRunningXacts(). Even if we add
!RecoveryInProgress() check there, the db-specific decoder on standbys
requires for the primary server to run a REPACK (CONCURRENTLY).
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 15:02 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-08 12:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-08 13:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-10 11:24 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Amit Kapila @ 2026-05-10 11:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Fri, May 8, 2026 at 7:28 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2026-May-08, Amit Kapila wrote:
>
> > Some issues/inefficiencies regarding this fix and base code related to
> > db-specific snapshots built during decoding: [...]
>
> Thanks for spending time reviewing this code. I think none of these
> problems are fundamental in nature, but they are obviously worth
> addressing for 19. If we hit some roadblock, we can still revert only
> db-specific snapshots.
>
IIUC, the emails by Andres [1][2] on db-specific snapshots sound like
concerns which are fundamental in nature. Apart from that as well, I
think the first point mentioned in my email [3] should be at least
addressed as that causes additional WAL even after reaching
consistent_state for each runing_xact record for a db-specific
decoder.
[1] - https://www.postgresql.org/message-id/cdgw4sbbfcgk6du3iv54r2dgiy4tfywoklbotlmj4irxavdcr3%40glxfw5jj2...
[2] - https://www.postgresql.org/message-id/pveffyxhnuurhb44uzqlwo3rkyzorkfh2rot7uwzlf2axhfvbp%407nrs2omys...
[3] - https://www.postgresql.org/message-id/CAA4eK1LygCDP3FiFzXY9iVNFcHxhf7TT_DFf7tryTu2oipmfpA%40mail.gma...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-10 11:31 ` Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-10 11:31 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, May 5, 2026 at 6:17 PM Antonin Houska <[email protected]> wrote:
>
> Antonin Houska <[email protected]> wrote:
>
> I think the problem is that with database-specific snapshot,
> SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
>
> /*
> * Database specific transaction info may exist to reach CONSISTENT state
> * faster, however the code below makes no use of it. Moreover, such
> * record might cause problems because the following normal (cluster-wide)
> * record can have lower value of oldestRunningXid. In that case, let's
> * wait with the cleanup for the next regular cluster-wide record.
> */
> if (OidIsValid(running->dbid))
> return;
>
> and thus some transactions whose XID is below running->oldestRunningXid may
> continue to be incorrectly considered running.
>
> I originally thought that this should not happen because such transactions
> will be added to the builder's array of committed transactions by
> SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
> a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> follow the xl_running_xacts record in WAL. In other words, even if
> xl_running_xacts is created before a COMMIT record of the contained
> transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> might not take place.
>
BTW, is it possible to write a test by using injection_points or via
manual steps (by using debugger, etc) so that we can more clearly
understand this problem and proposed fix?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-11 15:17 ` Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 02:25 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-05-11 15:17 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Amit Kapila <[email protected]> wrote:
> On Tue, May 5, 2026 at 6:17 PM Antonin Houska <[email protected]> wrote:
> >
> > Antonin Houska <[email protected]> wrote:
> >
> > I think the problem is that with database-specific snapshot,
> > SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
> >
> > /*
> > * Database specific transaction info may exist to reach CONSISTENT state
> > * faster, however the code below makes no use of it. Moreover, such
> > * record might cause problems because the following normal (cluster-wide)
> > * record can have lower value of oldestRunningXid. In that case, let's
> > * wait with the cleanup for the next regular cluster-wide record.
> > */
> > if (OidIsValid(running->dbid))
> > return;
> >
> > and thus some transactions whose XID is below running->oldestRunningXid may
> > continue to be incorrectly considered running.
> >
> > I originally thought that this should not happen because such transactions
> > will be added to the builder's array of committed transactions by
> > SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
> > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > follow the xl_running_xacts record in WAL. In other words, even if
> > xl_running_xacts is created before a COMMIT record of the contained
> > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > might not take place.
> >
>
> BTW, is it possible to write a test by using injection_points or via
> manual steps (by using debugger, etc) so that we can more clearly
> understand this problem and proposed fix?
So far I could observe the situation in WAL, but have no idea how it can
happen. For example, transaction 49242 gets committed here
rmgr: Transaction len (rec/tot): 46/ 46, tx: 49242, lsn: 0/18BC28C8, prev
0/18BC2890, desc: COMMIT 2026-05-11 16:38:16.603265 CEST
and then it appears in the 'xids' list of RUNNING_XACTS:
rmgr: Standby len (rec/tot): 106/ 106, tx: 0, lsn:
0/18BC3140, prev 0/18BC3100, desc: RUNNING_XACTS nextXid 49255
latestCompletedXid 49241 oldestRunningXid 49242; 13 xacts: 49248 49249 49246
49243 49252 49251 49244 49245 49242 49250 49253 49254 49247; dbid:5
I thought the situation is quite common (and therefore nothing of
SnapBuildProcessRunningXacts() should be skipped), but when trying to
reproduce the problem, I noticed that LogStandbySnapshot() shouldn't allow
that ordering issue when logical decoding is enabled:
/*
* GetRunningTransactionData() acquired ProcArrayLock, we must release it.
* For Hot Standby this can be done before inserting the WAL record
* because ProcArrayApplyRecoveryInfo() rechecks the commit status using
* the clog. For logical decoding, though, the lock can't be released
* early because the clog might be "in the future" from the POV of the
* historic snapshot. This would allow for situations where we're waiting
* for the end of a transaction listed in the xl_running_xacts record
* which, according to the WAL, has committed before the xl_running_xacts
* record. Fortunately this routine isn't executed frequently, and it's
* only a shared lock.
*/
if (!logical_decoding_enabled)
LWLockRelease(ProcArrayLock);
So I don't have the answer right now.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-11 19:30 ` Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-05-11 19:30 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Antonin Houska <[email protected]> wrote:
> Amit Kapila <[email protected]> wrote:
>
> > On Tue, May 5, 2026 at 6:17 PM Antonin Houska <[email protected]> wrote:
> > >
> > > Antonin Houska <[email protected]> wrote:
> > >
> > > I think the problem is that with database-specific snapshot,
> > > SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
> > >
> > > /*
> > > * Database specific transaction info may exist to reach CONSISTENT state
> > > * faster, however the code below makes no use of it. Moreover, such
> > > * record might cause problems because the following normal (cluster-wide)
> > > * record can have lower value of oldestRunningXid. In that case, let's
> > > * wait with the cleanup for the next regular cluster-wide record.
> > > */
> > > if (OidIsValid(running->dbid))
> > > return;
> > >
> > > and thus some transactions whose XID is below running->oldestRunningXid may
> > > continue to be incorrectly considered running.
> > >
> > > I originally thought that this should not happen because such transactions
> > > will be added to the builder's array of committed transactions by
> > > SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
> > > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > > follow the xl_running_xacts record in WAL. In other words, even if
> > > xl_running_xacts is created before a COMMIT record of the contained
> > > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > > might not take place.
> > >
> >
> > BTW, is it possible to write a test by using injection_points or via
> > manual steps (by using debugger, etc) so that we can more clearly
> > understand this problem and proposed fix?
>
> So far I could observe the situation in WAL, but have no idea how it can
> happen. For example, transaction 49242 gets committed here
>
> rmgr: Transaction len (rec/tot): 46/ 46, tx: 49242, lsn: 0/18BC28C8, prev
> 0/18BC2890, desc: COMMIT 2026-05-11 16:38:16.603265 CEST
>
> and then it appears in the 'xids' list of RUNNING_XACTS:
>
> rmgr: Standby len (rec/tot): 106/ 106, tx: 0, lsn:
> 0/18BC3140, prev 0/18BC3100, desc: RUNNING_XACTS nextXid 49255
> latestCompletedXid 49241 oldestRunningXid 49242; 13 xacts: 49248 49249 49246
> 49243 49252 49251 49244 49245 49242 49250 49253 49254 49247; dbid:5
>
>
> I thought the situation is quite common (and therefore nothing of
> SnapBuildProcessRunningXacts() should be skipped), but when trying to
> reproduce the problem, I noticed that LogStandbySnapshot() shouldn't allow
> that ordering issue when logical decoding is enabled:
>
> /*
> * GetRunningTransactionData() acquired ProcArrayLock, we must release it.
> * For Hot Standby this can be done before inserting the WAL record
> * because ProcArrayApplyRecoveryInfo() rechecks the commit status using
> * the clog. For logical decoding, though, the lock can't be released
> * early because the clog might be "in the future" from the POV of the
> * historic snapshot. This would allow for situations where we're waiting
> * for the end of a transaction listed in the xl_running_xacts record
> * which, according to the WAL, has committed before the xl_running_xacts
> * record. Fortunately this routine isn't executed frequently, and it's
> * only a shared lock.
> */
> if (!logical_decoding_enabled)
> LWLockRelease(ProcArrayLock);
>
> So I don't have the answer right now.
I think now that "waiting for the end of a transaction listed in the
xl_running_xacts record" in the comment is about transaction removal from
procarray. However, the COMMIT record can still be ahead of xl_running_xacts
because RecordTransactionCommit() is called before
ProcArrayEndTransaction(). I'll think again if the whole problem can be
reproduced with injection points.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-12 07:57 ` Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-12 07:57 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, May 12, 2026 at 1:00 AM Antonin Houska <[email protected]> wrote:
>
> Antonin Houska <[email protected]> wrote:
>
> > Amit Kapila <[email protected]> wrote:
> >
> > > On Tue, May 5, 2026 at 6:17 PM Antonin Houska <[email protected]> wrote:
> > > >
> > > > Antonin Houska <[email protected]> wrote:
> > > >
> > > > I think the problem is that with database-specific snapshot,
> > > > SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
> > > >
> > > > /*
> > > > * Database specific transaction info may exist to reach CONSISTENT state
> > > > * faster, however the code below makes no use of it. Moreover, such
> > > > * record might cause problems because the following normal (cluster-wide)
> > > > * record can have lower value of oldestRunningXid. In that case, let's
> > > > * wait with the cleanup for the next regular cluster-wide record.
> > > > */
> > > > if (OidIsValid(running->dbid))
> > > > return;
> > > >
> > > > and thus some transactions whose XID is below running->oldestRunningXid may
> > > > continue to be incorrectly considered running.
> > > >
> > > > I originally thought that this should not happen because such transactions
> > > > will be added to the builder's array of committed transactions by
> > > > SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
> > > > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > > > follow the xl_running_xacts record in WAL. In other words, even if
> > > > xl_running_xacts is created before a COMMIT record of the contained
> > > > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > > > might not take place.
> > > >
> > >
> > > BTW, is it possible to write a test by using injection_points or via
> > > manual steps (by using debugger, etc) so that we can more clearly
> > > understand this problem and proposed fix?
> >
> > So far I could observe the situation in WAL, but have no idea how it can
> > happen. For example, transaction 49242 gets committed here
> >
> > rmgr: Transaction len (rec/tot): 46/ 46, tx: 49242, lsn: 0/18BC28C8, prev
> > 0/18BC2890, desc: COMMIT 2026-05-11 16:38:16.603265 CEST
> >
> > and then it appears in the 'xids' list of RUNNING_XACTS:
> >
> > rmgr: Standby len (rec/tot): 106/ 106, tx: 0, lsn:
> > 0/18BC3140, prev 0/18BC3100, desc: RUNNING_XACTS nextXid 49255
> > latestCompletedXid 49241 oldestRunningXid 49242; 13 xacts: 49248 49249 49246
> > 49243 49252 49251 49244 49245 49242 49250 49253 49254 49247; dbid:5
> >
> >
> > I thought the situation is quite common (and therefore nothing of
> > SnapBuildProcessRunningXacts() should be skipped), but when trying to
> > reproduce the problem, I noticed that LogStandbySnapshot() shouldn't allow
> > that ordering issue when logical decoding is enabled:
> >
> > /*
> > * GetRunningTransactionData() acquired ProcArrayLock, we must release it.
> > * For Hot Standby this can be done before inserting the WAL record
> > * because ProcArrayApplyRecoveryInfo() rechecks the commit status using
> > * the clog. For logical decoding, though, the lock can't be released
> > * early because the clog might be "in the future" from the POV of the
> > * historic snapshot. This would allow for situations where we're waiting
> > * for the end of a transaction listed in the xl_running_xacts record
> > * which, according to the WAL, has committed before the xl_running_xacts
> > * record. Fortunately this routine isn't executed frequently, and it's
> > * only a shared lock.
> > */
> > if (!logical_decoding_enabled)
> > LWLockRelease(ProcArrayLock);
> >
> > So I don't have the answer right now.
>
> I think now that "waiting for the end of a transaction listed in the
> xl_running_xacts record" in the comment is about transaction removal from
> procarray. However, the COMMIT record can still be ahead of xl_running_xacts
> because RecordTransactionCommit() is called before
> ProcArrayEndTransaction().
>
I see your point. Due to this, once the xmin regresses based on
cluster-wide running_xact, some transaction could appear to be running
when it should have appeared as committed. However, still it is not
clear how it could lead to one update in the transaction as
successfully decoded and another one to be skipped. One theory could
be that before the second update, somehow invalidation happens and
when decoding tries to reload the catalog to decode second update, the
relation is not visible because xmin has regressed and the update is
somehow skipped. I can't see how it can happen in code but something
like that is happening. Assuming, the problematic case is something
like what I described, even than the fix of skipping cluster-wide
running xacts and instead LOG db-specific running_xacts to help
updating builder's xmin sounds inelegant and probably inefficient. For
example, I think such a dependency means we can never enable
db-specific snapshots on standby.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-12 11:08 ` Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-05-12 11:08 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Amit Kapila <[email protected]> wrote:
> On Tue, May 12, 2026 at 1:00 AM Antonin Houska <[email protected]> wrote:
> >
> > Antonin Houska <[email protected]> wrote:
> >
> > > Amit Kapila <[email protected]> wrote:
> > >
> > > > On Tue, May 5, 2026 at 6:17 PM Antonin Houska <[email protected]> wrote:
> > > > >
> > > > > Antonin Houska <[email protected]> wrote:
> > > > >
> > > > > I think the problem is that with database-specific snapshot,
> > > > > SnapBuildProcessRunningXacts() returns early, w/o adjusting builder->xmin
> > > > >
> > > > > /*
> > > > > * Database specific transaction info may exist to reach CONSISTENT state
> > > > > * faster, however the code below makes no use of it. Moreover, such
> > > > > * record might cause problems because the following normal (cluster-wide)
> > > > > * record can have lower value of oldestRunningXid. In that case, let's
> > > > > * wait with the cleanup for the next regular cluster-wide record.
> > > > > */
> > > > > if (OidIsValid(running->dbid))
> > > > > return;
> > > > >
> > > > > and thus some transactions whose XID is below running->oldestRunningXid may
> > > > > continue to be incorrectly considered running.
> > > > >
> > > > > I originally thought that this should not happen because such transactions
> > > > > will be added to the builder's array of committed transactions by
> > > > > SnapBuildCommitTxn() anyway. However, I failed to notice that COMMIT record of
> > > > > a transaction listed in the xl_running_xacts WAL record is not guaranteed to
> > > > > follow the xl_running_xacts record in WAL. In other words, even if
> > > > > xl_running_xacts is created before a COMMIT record of the contained
> > > > > transaction, it may end up at higher LSN in WAL. So the cleanup I relied on
> > > > > might not take place.
> > > > >
> > > >
> > > > BTW, is it possible to write a test by using injection_points or via
> > > > manual steps (by using debugger, etc) so that we can more clearly
> > > > understand this problem and proposed fix?
> > >
> > > So far I could observe the situation in WAL, but have no idea how it can
> > > happen. For example, transaction 49242 gets committed here
> > >
> > > rmgr: Transaction len (rec/tot): 46/ 46, tx: 49242, lsn: 0/18BC28C8, prev
> > > 0/18BC2890, desc: COMMIT 2026-05-11 16:38:16.603265 CEST
> > >
> > > and then it appears in the 'xids' list of RUNNING_XACTS:
> > >
> > > rmgr: Standby len (rec/tot): 106/ 106, tx: 0, lsn:
> > > 0/18BC3140, prev 0/18BC3100, desc: RUNNING_XACTS nextXid 49255
> > > latestCompletedXid 49241 oldestRunningXid 49242; 13 xacts: 49248 49249 49246
> > > 49243 49252 49251 49244 49245 49242 49250 49253 49254 49247; dbid:5
> > >
> > >
> > > I thought the situation is quite common (and therefore nothing of
> > > SnapBuildProcessRunningXacts() should be skipped), but when trying to
> > > reproduce the problem, I noticed that LogStandbySnapshot() shouldn't allow
> > > that ordering issue when logical decoding is enabled:
> > >
> > > /*
> > > * GetRunningTransactionData() acquired ProcArrayLock, we must release it.
> > > * For Hot Standby this can be done before inserting the WAL record
> > > * because ProcArrayApplyRecoveryInfo() rechecks the commit status using
> > > * the clog. For logical decoding, though, the lock can't be released
> > > * early because the clog might be "in the future" from the POV of the
> > > * historic snapshot. This would allow for situations where we're waiting
> > > * for the end of a transaction listed in the xl_running_xacts record
> > > * which, according to the WAL, has committed before the xl_running_xacts
> > > * record. Fortunately this routine isn't executed frequently, and it's
> > > * only a shared lock.
> > > */
> > > if (!logical_decoding_enabled)
> > > LWLockRelease(ProcArrayLock);
> > >
> > > So I don't have the answer right now.
> >
> > I think now that "waiting for the end of a transaction listed in the
> > xl_running_xacts record" in the comment is about transaction removal from
> > procarray. However, the COMMIT record can still be ahead of xl_running_xacts
> > because RecordTransactionCommit() is called before
> > ProcArrayEndTransaction().
> >
>
> I see your point. Due to this, once the xmin regresses based on
> cluster-wide running_xact, some transaction could appear to be running
> when it should have appeared as committed.
The problem is that xmin does not advance when it should. Attached is a test
that reproduces the problem (it includes [1], to handle injection points in
background worker), I hope the comments in the specification file are helpful.
It's actually not exactly the problem reported in the stress test, but IMO the
core issue is the same: effects of some transactions are lost. In the stress
test, tuple deletion was lost, so the error was "could not create unique
index". Here I only demonstrate lost INSERT.
> Assuming, the problematic case is something
> like what I described, even than the fix of skipping cluster-wide
> running xacts and instead LOG db-specific running_xacts to help
> updating builder's xmin sounds inelegant and probably inefficient. For
> example, I think such a dependency means we can never enable
> db-specific snapshots on standby.
ok
[1] https://www.postgresql.org/message-id/4703.1774250534%40localhost
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-13 12:34 ` Amit Kapila <[email protected]>
2026-05-13 16:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-13 12:34 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, May 12, 2026 at 4:38 PM Antonin Houska <[email protected]> wrote:
>
> Amit Kapila <[email protected]> wrote:
>
> >
> > I see your point. Due to this, once the xmin regresses based on
> > cluster-wide running_xact, some transaction could appear to be running
> > when it should have appeared as committed.
>
> The problem is that xmin does not advance when it should. Attached is a test
> that reproduces the problem (it includes [1], to handle injection points in
> background worker), I hope the comments in the specification file are helpful.
>
Yes, the comments were helpful. IIUC, the test skipped insert into
repack_test because the transaction doing that insert happened before
the snapbuild reached FULL_SNAPSHOT/CONSISTENT state, so its commit is
not decoded. Then we also didn't update builder->xmin after reaching
CONSISTENT state in the last running_xact record for MyDatabase. So,
the insert is neither covered in initial copy, nor decoded, so after
repack (concurrently) is finished the table is empty.
I think the patch proposed will fix this specific issue but apart from
other points raised for this patch few more are: (a) Post CONSISTENT
state, for cleanup of db_specific snapshots, we need to separately
again LOG db-specific running xacts whenever we encounter another
running_xacts, (b) Other point is that because
repack-concurrently always use full-snapshot, the serialization of
snapshot is useless because it can't be used by others.
> It's actually not exactly the problem reported in the stress test, but IMO the
> core issue is the same: effects of some transactions are lost. In the stress
> test, tuple deletion was lost, so the error was "could not create unique
> index". Here I only demonstrate lost INSERT.
>
> > Assuming, the problematic case is something
> > like what I described, even than the fix of skipping cluster-wide
> > running xacts and instead LOG db-specific running_xacts to help
> > updating builder's xmin sounds inelegant and probably inefficient. For
> > example, I think such a dependency means we can never enable
> > db-specific snapshots on standby.
>
> ok
>
So now the question is where do we go from here. I am not confident
that the current code to achieve db-specific snapshots in logical
decoding is the best possible solution both because of the drawbacks
(like we won't be able to enable this on standby) and inefficiencies
pointed out by me in this and previous emails in this work.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-13 16:58 ` Alvaro Herrera <[email protected]>
2026-05-14 07:02 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-13 16:58 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello Amit,
On 2026-May-13, Amit Kapila wrote:
> So now the question is where do we go from here. I am not confident
> that the current code to achieve db-specific snapshots in logical
> decoding is the best possible solution both because of the drawbacks
> (like we won't be able to enable this on standby) and inefficiencies
> pointed out by me in this and previous emails in this work.
This is a fair question. I don't think we have time to go much further
on this aspect before beta 1, so we either accept this patch, fix the
inefficiencies you pointed out and keep db-specific snapshots, or we
revert db-specific snapshots and go back to the standard snapshot-taking
technique for REPACK in 19 and see what we can improve for 20.
Now, the worst consequence of reverting db-specific snapshots is that
you will only be able to run REPACK in a single database at a time
(because any subsequent REPACK will have to wait until the first one
finishes before being able to get its snapshot). In most normal cases
this is probably not a big deal. But if you have a multitenant system,
and you want your users to be able to run REPACK on their tables, you
may be a bit screwed. So I hesitate to just go and revert it without
offering those people any alternative.
(It's also possible that being unable to run more than one REPACK at a
time is not so big a deal. After all, it's supposed to be an infrequent
operation. And users probably don't or shouldn't have multi-terabyte
tables in multitenant databases anyway.)
I'm not sure I understand the point of the standby. I mean, you can't
run REPACK on the standby anyway, so I don't see this as a very
problematic restriction. Do you have other reasons for wanting a
db-specific snapshot in a standby?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-13 16:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-14 07:02 ` Amit Kapila <[email protected]>
2026-05-19 18:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-14 07:02 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, May 13, 2026 at 10:28 PM Alvaro Herrera <[email protected]> wrote:
>
> Hello Amit,
>
> On 2026-May-13, Amit Kapila wrote:
>
> > So now the question is where do we go from here. I am not confident
> > that the current code to achieve db-specific snapshots in logical
> > decoding is the best possible solution both because of the drawbacks
> > (like we won't be able to enable this on standby) and inefficiencies
> > pointed out by me in this and previous emails in this work.
>
> This is a fair question. I don't think we have time to go much further
> on this aspect before beta 1, so we either accept this patch, fix the
> inefficiencies you pointed out and keep db-specific snapshots,
>
I don't think it would be easy to address these inefficiencies before
beta 1. The root of those inefficiencies is that the patch reuses the
cluster-wide running_xact WAL infrastructure to log db-specific
running transactions, and then tries to feed that into the existing
snapbuild machinery to reach a consistent state.
As another example of this mismatch that occurred to me today: in
SnapBuildCommitTxn, we are tracking the committed_xip array for all
cluster-wide XIDs, even when using a db-specific snapshot. A
db-specific snapshot shouldn't need to care about XIDs from other
databases. We only try to take care of it in one part of the system
where process running_xacts record. I admit that I don't know at this
stage what exactly we should do about it but all such things deserve a
discussion and careful thought.
The broader issue is that the entire logical decoding mechanism is
designed to process cluster-wide transactions. This patch tries to
bypass that foundational assumption, but only during the initial
snapshot construction while processing running_xacts record.
To be clear, I am not against the idea of db-specific snapshots to
enable concurrent repacks. My concern is simply the time required to
get the architecture right. In its current state, we need more time to
carefully consider how this db-specific concept interacts with the
rest of the logical decoding machinery, which is built for
cluster-wide records.
> or we
> revert db-specific snapshots and go back to the standard snapshot-taking
> technique for REPACK in 19 and see what we can improve for 20.
>
> Now, the worst consequence of reverting db-specific snapshots is that
> you will only be able to run REPACK in a single database at a time
> (because any subsequent REPACK will have to wait until the first one
> finishes before being able to get its snapshot). In most normal cases
> this is probably not a big deal. But if you have a multitenant system,
> and you want your users to be able to run REPACK on their tables, you
> may be a bit screwed. So I hesitate to just go and revert it without
> offering those people any alternative.
>
I understand your point but I feel we can extend the current feature
in future versions to address such cases (allow REPACK CONCURRENTLY on
tables in multiple-databases simultaneously). For now, they may need
to rely on REPACK without CONCURRENTLY option, if they want to use it
for multiple databases simultaneously.
> (It's also possible that being unable to run more than one REPACK at a
> time is not so big a deal. After all, it's supposed to be an infrequent
> operation. And users probably don't or shouldn't have multi-terabyte
> tables in multitenant databases anyway.)
>
> I'm not sure I understand the point of the standby. I mean, you can't
> run REPACK on the standby anyway, so I don't see this as a very
> problematic restriction. Do you have other reasons for wanting a
> db-specific snapshot in a standby?
>
We are exposing need_shared_catalogs as a generic plugin option,
defined as: 'it can be set to false if one is certain the plugin
functions do not access shared system catalogs.' This implies it can
be used for purposes other than REPACK.
For example, one can imagine a single-database audit plugin that only
cares about data modifications within a specific database. By setting
need_shared_catalogs = false on a standby, it could reach a CONSISTENT
state much faster, perfectly serving its needs.
While such a plugin might not exist right now, my broader point is
this: when we expose a generic facility, it can and will be used in
ways beyond our initial core use cases. We should try to ensure the
design doesn't permanently preclude such extensions. With the current
design choice, we are painting ourselves into a corner where this
feature cannot easily be extended to standbys even in the future.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-13 16:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-14 07:02 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-19 18:52 ` Alvaro Herrera <[email protected]>
2026-05-23 15:29 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-19 18:52 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-14, Amit Kapila wrote:
> The broader issue is that the entire logical decoding mechanism is
> designed to process cluster-wide transactions. This patch tries to
> bypass that foundational assumption, but only during the initial
> snapshot construction while processing running_xacts record.
>
> To be clear, I am not against the idea of db-specific snapshots to
> enable concurrent repacks. My concern is simply the time required to
> get the architecture right. In its current state, we need more time to
> carefully consider how this db-specific concept interacts with the
> rest of the logical decoding machinery, which is built for
> cluster-wide records.
Hmm. So at this point I have to admit that the time I'll have before
beta 1 is going to be very scarce. You're probably right that it's
better to revert db-specific snapshots in pg19, and try again for 20.
The revert should be a simple patch.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-13 16:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-14 07:02 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-19 18:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-05-23 15:29 ` Amit Kapila <[email protected]>
2026-05-24 11:29 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-05-23 15:29 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, May 19, 2026 at 11:52 AM Alvaro Herrera <[email protected]> wrote:
>
> On 2026-May-14, Amit Kapila wrote:
>
> > The broader issue is that the entire logical decoding mechanism is
> > designed to process cluster-wide transactions. This patch tries to
> > bypass that foundational assumption, but only during the initial
> > snapshot construction while processing running_xacts record.
> >
> > To be clear, I am not against the idea of db-specific snapshots to
> > enable concurrent repacks. My concern is simply the time required to
> > get the architecture right. In its current state, we need more time to
> > carefully consider how this db-specific concept interacts with the
> > rest of the logical decoding machinery, which is built for
> > cluster-wide records.
>
> Hmm. So at this point I have to admit that the time I'll have before
> beta 1 is going to be very scarce. You're probably right that it's
> better to revert db-specific snapshots in pg19, and try again for 20.
>
Sounds reasonable.
> The revert should be a simple patch.
>
I also think so.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-11 19:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-12 07:57 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-12 11:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-13 12:34 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-13 16:58 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-14 07:02 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-19 18:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-05-23 15:29 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-24 11:29 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-24 11:29 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Antonin Houska <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-May-23, Amit Kapila wrote:
> > The revert should be a simple patch.
>
> I also think so.
Okay, pushed the revert after seeing it pass CI:
https://cirrus-ci.com/build/5520722497372160
Thanks!
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-05 12:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-10 11:31 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-05-11 15:17 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-05-12 02:25 ` Hayato Kuroda (Fujitsu) <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-05-12 02:25 UTC (permalink / raw)
To: 'Antonin Houska' <[email protected]>; Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Dear Antonin,
FYI, I have also been spending time to reproduce the failure but I have not done yet.
> So far I could observe the situation in WAL, but have no idea how it can
> happen.
Not sure it matches with your situation, but I could reproduce the situation by
using gdb.
0. initialized an instance with wal_level=logical and defined a table.
1. established a connection
2. attached the backend via gdb
3. added a breakpoint in ProcArrayEndTransaction
4. committed a transaction, and it would stop at the breakpoint
5. established another connection
6. ran REPACK CONCURRENTLY
7. detached from the first backend.
8. all commands would finish.
This could allow that COMMIT record exists ahead the RUNNING_XACTS record.
When the backend reaches CommitTransaction()->ProcArrayEndTransaction(), the commit
record has already been serialized, but the transaction is still marked as active
on the proc array. Above workload allowed that repack worker could check in-between.
[1]:
```
rmgr: Transaction len (rec/tot): 46/ 46, tx: 695, lsn: 0/018B89C0, prev 0/018B8980, desc: COMMIT 2026-05-12 11:11:31.588061 JST
rmgr: Standby len (rec/tot): 58/ 58, tx: 0, lsn: 0/018B89F0, prev 0/018B89C0, desc: RUNNING_XACTS nextXid 696 latestCompletedXid 694 oldestRunningXid 695; 1 xacts: 695; dbid: 0
rmgr: Standby len (rec/tot): 58/ 58, tx: 0, lsn: 0/018B8A30, prev 0/018B89F0, desc: RUNNING_XACTS nextXid 696 latestCompletedXid 694 oldestRunningXid 695; 1 xacts: 695; dbid: 5
```
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:15 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-07 12:33 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-07 15:38 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-07 15:48 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-27 04:25 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-30 11:24 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-05-05 15:04 ` Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-05-05 15:04 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On 2026-Apr-30, Mihail Nikalayeu wrote:
> P.S.
> I think it is good idea to add these stress tests to the source tree,
> perhaps with some kind PG_TEST_EXTRA=stress (as done in [1]).
Yeah, agreed. I'll see to that shortly.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I am amazed at [the pgsql-sql] mailing list for the wonderful support, and
lack of hesitasion in answering a lost soul's question, I just wished the rest
of the mailing list could be like this." (Fotis)
https://postgr.es/m/[email protected]
^ permalink raw reply [nested|flat] 156+ messages in thread
* RE: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 12:32 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-07 14:19 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-05-01 07:30 ` Re: Adding REPACK [concurrently] 'Alvaro Herrera' <[email protected]>
4 siblings, 2 replies; 156+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-04-07 12:32 UTC (permalink / raw)
To: 'Alvaro Herrera' <[email protected]>; Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Dear Álvaro,
Thanks for updating the patch. I have two comments/questions.
01.
```
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -394,6 +394,14 @@ systable_beginscan(Relation heapRelation,
SysScanDesc sysscan;
Relation irel;
+ /*
+ * If this backend promised that it won't access shared catalogs during
+ * logical decoding, this it the right place to verify.
+ */
+ Assert(!HistoricSnapshotActive() ||
+ accessSharedCatalogsInDecoding ||
+ !heapRelation->rd_rel->relisshared);
```
Not sure it's OK to use Assert(). elog(ERROR) might be better if we want to really
avoid the case.
02. SnapBuildProcessRunningXacts
Per my understanding, the db_specic snapshot can be also serialized. Is it
possibility tha normal logical decoding system restores the snapshot and obtain
the wrong result?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:32 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
@ 2026-04-07 14:19 ` Antonin Houska <[email protected]>
2026-04-08 03:49 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-07 14:19 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: 'Alvaro Herrera' <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hayato Kuroda (Fujitsu) <[email protected]> wrote:
> 02. SnapBuildProcessRunningXacts
>
> Per my understanding, the db_specic snapshot can be also serialized. Is it
> possibility tha normal logical decoding system restores the snapshot and obtain
> the wrong result?
I don't think that the database-specific xl_running_xacts WAL record affects
what SnapBuildSerialize() writes to disk: the contents of builder->committed,
etc. is updated by decoding COMMIT and ABORT records.
On the other hand, with that kind of record, there's probably no reason to
call SnapBuildSerialize(). Good catch, thanks.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:32 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-07 14:19 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-08 03:49 ` Amit Kapila <[email protected]>
2026-05-08 23:22 ` Re: Adding REPACK [concurrently] Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-04-08 03:49 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, Apr 7, 2026 at 7:49 PM Antonin Houska <[email protected]> wrote:
>
> Hayato Kuroda (Fujitsu) <[email protected]> wrote:
>
> > 02. SnapBuildProcessRunningXacts
> >
> > Per my understanding, the db_specic snapshot can be also serialized. Is it
> > possibility tha normal logical decoding system restores the snapshot and obtain
> > the wrong result?
>
> I don't think that the database-specific xl_running_xacts WAL record affects
> what SnapBuildSerialize() writes to disk: the contents of builder->committed,
> etc. is updated by decoding COMMIT and ABORT records.
>
I think the point is that the other process say a walsender could
restore such a snapshot making it take the wrong decision.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:32 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-07 14:19 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-08 03:49 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-05-08 23:22 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Masahiko Sawada @ 2026-05-08 23:22 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Antonin Houska <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Tue, Apr 7, 2026 at 8:49 PM Amit Kapila <[email protected]> wrote:
>
> On Tue, Apr 7, 2026 at 7:49 PM Antonin Houska <[email protected]> wrote:
> >
> > Hayato Kuroda (Fujitsu) <[email protected]> wrote:
> >
> > > 02. SnapBuildProcessRunningXacts
> > >
> > > Per my understanding, the db_specic snapshot can be also serialized. Is it
> > > possibility tha normal logical decoding system restores the snapshot and obtain
> > > the wrong result?
> >
> > I don't think that the database-specific xl_running_xacts WAL record affects
> > what SnapBuildSerialize() writes to disk: the contents of builder->committed,
> > etc. is updated by decoding COMMIT and ABORT records.
> >
>
> I think the point is that the other process say a walsender could
> restore such a snapshot making it take the wrong decision.
>
Right. I think it affects even concurrent REPACK (CONCURRENTLY)
running on other databases. They could end up restoring the snapshot
serialized by another REPACK command running on another database and
becoming the consistent state without waiting for transactions
concurrently running on the same database, which is clearly wrong.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 12:32 ` RE: Adding REPACK [concurrently] Hayato Kuroda (Fujitsu) <[email protected]>
@ 2026-05-01 07:30 ` 'Alvaro Herrera' <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: 'Alvaro Herrera' @ 2026-05-01 07:30 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello,
On 2026-Apr-07, Hayato Kuroda (Fujitsu) wrote:
> 01.
> ```
> --- a/src/backend/access/index/genam.c
> +++ b/src/backend/access/index/genam.c
> @@ -394,6 +394,14 @@ systable_beginscan(Relation heapRelation,
> SysScanDesc sysscan;
> Relation irel;
>
> + /*
> + * If this backend promised that it won't access shared catalogs during
> + * logical decoding, this it the right place to verify.
> + */
> + Assert(!HistoricSnapshotActive() ||
> + accessSharedCatalogsInDecoding ||
> + !heapRelation->rd_rel->relisshared);
> ```
>
> Not sure it's OK to use Assert(). elog(ERROR) might be better if we want to really
> avoid the case.
How about the attached?
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 19:43 ` Robert Treat <[email protected]>
2026-04-08 07:31 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
4 siblings, 1 reply; 156+ messages in thread
From: Robert Treat @ 2026-04-07 19:43 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>
On Mon, Apr 6, 2026 at 6:22 PM Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-06, Mihail Nikalayeu wrote:
<snip>
>
> Anyway, here's the three missing parts. I have not yet edited the
> deadlock-checker one to protect autovacuum from processing tables under
> repack.
>
I have this lingering bit of paranoia that users could end up in a
situation with a large / long running repack that goes past failsafe
age which prevents the simpler fix of failsafe autovacuum from
running. While the repack finishing would resolve this issue, we can't
know ahead of time that the repack would finish in time, and
statistically speaking, failsafe autovacuum should generally run much
quicker than any repack could. I'm not sure if that means we should
let failsafe vacuum cancel repacks (that seems a bit extreme), but
maybe we want to help $operator to think about this decision, except
if we don't allow autovacuum to wait and we don't allow it to respawn,
I wonder if the end user will ever realize they are in this position.
Granted, there doesn't seem like a clean fix for this...
Robert Treat
https://xzilla.net
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 19:43 ` Re: Adding REPACK [concurrently] Robert Treat <[email protected]>
@ 2026-04-08 07:31 ` Antonin Houska <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-08 07:31 UTC (permalink / raw)
To: Robert Treat <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>
Robert Treat <[email protected]> wrote:
> On Mon, Apr 6, 2026 at 6:22 PM Alvaro Herrera <[email protected]> wrote:
> > On 2026-Apr-06, Mihail Nikalayeu wrote:
> <snip>
> >
> > Anyway, here's the three missing parts. I have not yet edited the
> > deadlock-checker one to protect autovacuum from processing tables under
> > repack.
> >
>
> I have this lingering bit of paranoia that users could end up in a
> situation with a large / long running repack that goes past failsafe
> age which prevents the simpler fix of failsafe autovacuum from
> running. While the repack finishing would resolve this issue, we can't
> know ahead of time that the repack would finish in time, and
> statistically speaking, failsafe autovacuum should generally run much
> quicker than any repack could. I'm not sure if that means we should
> let failsafe vacuum cancel repacks (that seems a bit extreme), but
> maybe we want to help $operator to think about this decision, except
> if we don't allow autovacuum to wait and we don't allow it to respawn,
> I wonder if the end user will ever realize they are in this position.
> Granted, there doesn't seem like a clean fix for this...
If REPACK is not going to finish in time, I think it makes little difference
whether VACUUM is allowed to wait or not: even if it waits, it will start just
too late. One reason to avoid waiting might be to allow autovacuum to work on
other tables in between.
I agree that the DBA should have some guidance to asses whether REPACK or
(failsafe) VACUUM is the appropriate action. While failsafe VACUUM is clearly
a means to avoid XID wraparound, I tend to consider REPACK primarily a command
to remove table bloat. Or is there a situation where REPACK is better even to
avoid the wraparound?
Technically, the deadlock can be avoided by not running DDLs on the table
while REPACK is running. I'm just thinking if, by mentioning this in the
REPACK documentation, we'd admit that the REPACK (CONCURRENTLY) feature is
actually incomplete. On the other hand, if we don't mention the risk of
deadlock, it's a similar situation to not mentioning it for commands like
ALTER TABLE: if ALTER TABLE performs table rewrite, deadlock can also result
in a significant amount of wasted resources. (Of course, it's not the same if
the purpose of REPACK is considered substitute for failsafe VACUUM, but I'm
not sure about that.)
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-07 20:24 ` Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
4 siblings, 1 reply; 156+ messages in thread
From: Andres Freund @ 2026-04-07 20:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-07 00:22:32 +0200, Alvaro Herrera wrote:
> From 4303eea0a72408183f9f5afcf8d2801df20f8ffe Mon Sep 17 00:00:00 2001
> From: Antonin Houska <[email protected]>
> Date: Wed, 1 Apr 2026 17:35:47 +0200
> Subject: [PATCH v56 3/3] Error out any process that would block at REPACK
>
> Any process waiting on REPACK to release its lock would actually cause
> it to deadlock when it tries to upgrade its lock to AEL, losing all work
> done to that point. We avoid this by teaching the deadlock detector to
> raise an error when this condition is detected.
I'm rather doubtful that that is ok. Won't this make it basically unsafe to
use repack in way way too many situations? If sessions start erroring out,
rather than wait, because they briefly lock a relation or such it'll imo make
this rather unsafe in production.
ISTM that the proper fix here might be to allow repack to do a lock upgrade
that jumps to the front of the lock's wait queue. The lock upgrade is the
reason for the deadlock, right? And the reason it will often cause a deadlock
is that the repack will be at the tail, rather than the front of the wait
queue. So let's fix that.
Jumping the queue won't make the lock upgrade immediately, of course, but I
think it would fix the issue of the repack, rather than some other process,
getting cancelled?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-08 08:35 ` Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-04-08 08:35 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Wed, Apr 8, 2026 at 1:54 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2026-04-07 00:22:32 +0200, Alvaro Herrera wrote:
> > From 4303eea0a72408183f9f5afcf8d2801df20f8ffe Mon Sep 17 00:00:00 2001
> > From: Antonin Houska <[email protected]>
> > Date: Wed, 1 Apr 2026 17:35:47 +0200
> > Subject: [PATCH v56 3/3] Error out any process that would block at REPACK
> >
> > Any process waiting on REPACK to release its lock would actually cause
> > it to deadlock when it tries to upgrade its lock to AEL, losing all work
> > done to that point. We avoid this by teaching the deadlock detector to
> > raise an error when this condition is detected.
>
> I'm rather doubtful that that is ok.
>
Another possible idea is that after copying table_data to the new
table, we mark the old table as in_use_by_repack and release the
ShareUpdateExclusiveLock on the old table. Then the function
CheckTableNotInUse() should be updated to give an ERROR if the table
is marked as in_use_by_repack. Now, acquiring AEL by repack
(concurrently) should be safe because all concurrent DDLs should be
errored out due to flag in_use_by_repack. Can this address the problem
we are worried about the lock upgrade?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-08 17:22 ` Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Andres Freund @ 2026-04-08 17:22 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Mihail Nikalayeu <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-08 14:05:49 +0530, Amit Kapila wrote:
> On Wed, Apr 8, 2026 at 1:54 AM Andres Freund <[email protected]> wrote:
> > On 2026-04-07 00:22:32 +0200, Alvaro Herrera wrote:
> > > From 4303eea0a72408183f9f5afcf8d2801df20f8ffe Mon Sep 17 00:00:00 2001
> > > From: Antonin Houska <[email protected]>
> > > Date: Wed, 1 Apr 2026 17:35:47 +0200
> > > Subject: [PATCH v56 3/3] Error out any process that would block at REPACK
> > >
> > > Any process waiting on REPACK to release its lock would actually cause
> > > it to deadlock when it tries to upgrade its lock to AEL, losing all work
> > > done to that point. We avoid this by teaching the deadlock detector to
> > > raise an error when this condition is detected.
> >
> > I'm rather doubtful that that is ok.
> >
>
> Another possible idea is that after copying table_data to the new
> table, we mark the old table as in_use_by_repack and release the
> ShareUpdateExclusiveLock on the old table. Then the function
> CheckTableNotInUse() should be updated to give an ERROR if the table
> is marked as in_use_by_repack. Now, acquiring AEL by repack
> (concurrently) should be safe because all concurrent DDLs should be
> errored out due to flag in_use_by_repack. Can this address the problem
> we are worried about the lock upgrade?
I don't think this is a viable path. You need to prevent any further lock
acquisitions on the relation to be able to swap it, not just conflicting DDL.
And you need to wait for all pre-existing locks to have been released. That
doesn't really get easier by what you propose.
I don't think CheckTableNotInUse() would work anyway - don't we already hold
locks by the point we call it? And even if that were not the case, there are
several paths to locking relations that don't ever go anywhere near
CheckTableNotInUse().
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-08 23:36 ` Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 14:13 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-08 23:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello, Andres!
On Wed, Apr 8, 2026 at 7:22 PM Andres Freund <[email protected]> wrote:
> I don't think this is a viable path. You need to prevent any further lock
> acquisitions on the relation to be able to swap it, not just conflicting DDL.
AFAIU, Amit's main idea is that we currently upgrade the lock instead
of **releasing and re-acquiring** it because we fear DDL between those
actions.
DML actions are ok for us; REPACK will wait for them while getting
AEL. Later changes will be applied by REPACK backend while holding
AEL.
One more thing we may prevent from sneaking into that hole is a
VACUUM. It will not break anything, but will be huge waste of time and
resources.
> And you need to wait for all pre-existing locks to have been released. That
> doesn't really get easier by what you propose.
Do you mean locks from other sessions accessing the table? Is it done
automatically while waiting for AEL?
> I don't think CheckTableNotInUse() would work anyway - don't we already hold
> locks by the point we call it?
Yes, the DDL session already holds locks by the time
CheckTableNotInUse is called - but is that really the problem? They
will be released on error.
> And even if that were not the case, there are
> several paths to locking relations that don't ever go anywhere near
> CheckTableNotInUse().
But those aren't DDL, so they shouldn't be the problem (CREATE TRIGGER
might be - it seems to ignore CheckTableNotInUse, but perhaps it's
fine).
So, in my undeerstanding Amit's idea has two parts: "set flag and
release/re-acquire" + "use CheckTableNotInUse (or some place like
that) to check the flag and fail for DDL commands."
Reading your arguments again, they all seem valid under the assumption
that we still hold SUEL while requesting AEL.
I suspect you may have just skimmed past the 'release/re-acquire' part
- but more probably I missed something.
Best regards,
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-09 04:54 ` Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Amit Kapila @ 2026-04-09 04:54 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
On Thu, Apr 9, 2026 at 5:08 AM Mihail Nikalayeu
<[email protected]> wrote:
>
> On Wed, Apr 8, 2026 at 7:22 PM Andres Freund <[email protected]> wrote:
> > I don't think this is a viable path. You need to prevent any further lock
> > acquisitions on the relation to be able to swap it, not just conflicting DDL.
>
> AFAIU, Amit's main idea is that we currently upgrade the lock instead
> of **releasing and re-acquiring** it because we fear DDL between those
> actions.
> DML actions are ok for us; REPACK will wait for them while getting
> AEL. Later changes will be applied by REPACK backend while holding
> AEL.
>
Yes, this is exactly what I had in mind.
> One more thing we may prevent from sneaking into that hole is a
> VACUUM. It will not break anything, but will be huge waste of time and
> resources.
>
We can prevent other commands (if required) by checking the
rel_in_use_by_repack flag but I thought for the initial version it is
better to do what is minimally required.
> > And you need to wait for all pre-existing locks to have been released. That
> > doesn't really get easier by what you propose.
>
> Do you mean locks from other sessions accessing the table? Is it done
> automatically while waiting for AEL?
>
Right and this is already the case with the code where locks
not-conflicting with ShareUpdateExclusiveLock could be present before
we try to upgrade the lock. The point was we will not let DDL execute
on the table after the new flag (rel_in_use_by_repack) is set and we
released the ShareUpdateExclusiveLock.
> > I don't think CheckTableNotInUse() would work anyway - don't we already hold
> > locks by the point we call it?
>
> Yes, the DDL session already holds locks by the time
> CheckTableNotInUse is called - but is that really the problem? They
> will be released on error.
>
Yes, that is the key to solve this problem. Let me take an example to
explain this a bit more. Right now, the problem can happen in the
following kind of sequence.
Session-1:
REPACK (CONCURRENTLY) foo;
-- now say after the above command has acquired
ShareUpdateExclusiveLock and is doing the work of copying the table,
Session-2 did following actions.
Session-2:
Select * from foo; -- this is allowed
ALTER TABLE foo ADD COLUMN c2; -- this will blocked as Session-1
already has acquired ShareUpdateExclusiveLock
Session-1:
-- continues and tries to upgrade the lock to AEL. This leads to
deadlock ERROR in the current session doing REPACK (CONCURRENTLY).
Now, with the solution I proposed, because we will release
ShareUpdateExclusiveLock after setting a flag like
rel_in_use_by_repack, the ALTER TABLE in session-2 will succeed but
will error_out by CheckTableNotInUse(or a similar function that checks
rel_in_use_by_repack). Both with and without this solution, acquiring
AEL by REPACK (CONCURRENTLY) needs to wait concurrent locks like the
one for SELECT in above example that could have been acquired during
the time REPACKing had ShareUpdateExclusiveLock.
> > And even if that were not the case, there are
> > several paths to locking relations that don't ever go anywhere near
> > CheckTableNotInUse().
>
> But those aren't DDL, so they shouldn't be the problem (CREATE TRIGGER
> might be - it seems to ignore CheckTableNotInUse, but perhaps it's
> fine).
>
> So, in my undeerstanding Amit's idea has two parts: "set flag and
> release/re-acquire" + "use CheckTableNotInUse (or some place like
> that) to check the flag and fail for DDL commands."
>
True, this is the core of the idea.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
@ 2026-04-09 08:43 ` Antonin Houska <[email protected]>
2026-04-09 09:11 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-09 08:43 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Amit Kapila <[email protected]> wrote:
> On Thu, Apr 9, 2026 at 5:08 AM Mihail Nikalayeu
> <[email protected]> wrote:
> >
> > On Wed, Apr 8, 2026 at 7:22 PM Andres Freund <[email protected]> wrote:
> > > I don't think this is a viable path. You need to prevent any further lock
> > > acquisitions on the relation to be able to swap it, not just conflicting DDL.
> >
> > AFAIU, Amit's main idea is that we currently upgrade the lock instead
> > of **releasing and re-acquiring** it because we fear DDL between those
> > actions.
> > DML actions are ok for us; REPACK will wait for them while getting
> > AEL. Later changes will be applied by REPACK backend while holding
> > AEL.
> >
>
> Yes, this is exactly what I had in mind.
>
> > One more thing we may prevent from sneaking into that hole is a
> > VACUUM. It will not break anything, but will be huge waste of time and
> > resources.
> >
>
> We can prevent other commands (if required) by checking the
> rel_in_use_by_repack flag but I thought for the initial version it is
> better to do what is minimally required.
>
> > > And you need to wait for all pre-existing locks to have been released. That
> > > doesn't really get easier by what you propose.
> >
> > Do you mean locks from other sessions accessing the table? Is it done
> > automatically while waiting for AEL?
> >
>
> Right and this is already the case with the code where locks
> not-conflicting with ShareUpdateExclusiveLock could be present before
> we try to upgrade the lock. The point was we will not let DDL execute
> on the table after the new flag (rel_in_use_by_repack) is set and we
> released the ShareUpdateExclusiveLock.
>
> > > I don't think CheckTableNotInUse() would work anyway - don't we already hold
> > > locks by the point we call it?
> >
> > Yes, the DDL session already holds locks by the time
> > CheckTableNotInUse is called - but is that really the problem? They
> > will be released on error.
> >
>
> Yes, that is the key to solve this problem. Let me take an example to
> explain this a bit more. Right now, the problem can happen in the
> following kind of sequence.
>
> Session-1:
> REPACK (CONCURRENTLY) foo;
>
> -- now say after the above command has acquired
> ShareUpdateExclusiveLock and is doing the work of copying the table,
> Session-2 did following actions.
>
> Session-2:
> Select * from foo; -- this is allowed
> ALTER TABLE foo ADD COLUMN c2; -- this will blocked as Session-1
> already has acquired ShareUpdateExclusiveLock
>
> Session-1:
> -- continues and tries to upgrade the lock to AEL. This leads to
> deadlock ERROR in the current session doing REPACK (CONCURRENTLY).
>
> Now, with the solution I proposed, because we will release
> ShareUpdateExclusiveLock after setting a flag like
> rel_in_use_by_repack, the ALTER TABLE in session-2 will succeed but
> will error_out by CheckTableNotInUse(or a similar function that checks
> rel_in_use_by_repack). Both with and without this solution, acquiring
> AEL by REPACK (CONCURRENTLY) needs to wait concurrent locks like the
> one for SELECT in above example that could have been acquired during
> the time REPACKing had ShareUpdateExclusiveLock.
>
> > > And even if that were not the case, there are
> > > several paths to locking relations that don't ever go anywhere near
> > > CheckTableNotInUse().
> >
> > But those aren't DDL, so they shouldn't be the problem (CREATE TRIGGER
> > might be - it seems to ignore CheckTableNotInUse, but perhaps it's
> > fine).
> >
> > So, in my undeerstanding Amit's idea has two parts: "set flag and
> > release/re-acquire" + "use CheckTableNotInUse (or some place like
> > that) to check the flag and fail for DDL commands."
> >
>
> True, this is the core of the idea.
This approach LGTM when it comes to concurrent DDLs. However, consider REPACK
holding ShareUpdateExclusiveLock (SUEL) and VACUUM (w/o VACOPT_SKIP_LOCKED)
waiting for the same lock. Once REPACK releases its SUEL, VACUUM gets it and
processes the table, then REPACK finally gets AccessExclusiveLock (AEL) and
finishes too. Nothing went wrong from the data consistency POV, however the
VACUUM proably wasted a lot of resources, because REPACK does "more than
VACUUM". Furthermore, while REPACK was waiting for the AEL (and the VACUUM was
running), a *lot of* DMLs could have been executed on the table, and REPACK
will have to replay those while holding AEL. The point is that REPACK should
hold AEL for as short time as possible.
What Andres proposed (AFAIU) should help to avoid this problem because
REPACK's request for AEL would get in front of the VACUUM's request for SUEL
in the queue. Of course, VACUUM would eventually run too, but - if REPACK
succeeded - it'd be much cheaper because it would (supposedly) find very
little work to do.
Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it should
possibly have higher priority than REPACK), but I think this prioritization
should be implemented in other way than just letting it get in the way of
REPACK (at the time REPACK is nearly finished).
[1] https://www.postgresql.org/message-id/CABV9wwMrrP4S54jPGn5D-a8AbJm%2Be5%2BWORs6ykUBgXdc-%2B%2BNtQ%40...
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-09 09:11 ` Mihail Nikalayeu <[email protected]>
2026-04-09 09:26 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-09 09:11 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi!
On Thu, Apr 9, 2026 at 10:43 AM Antonin Houska <[email protected]> wrote:
> This approach LGTM when it comes to concurrent DDLs. However, consider REPACK
> holding ShareUpdateExclusiveLock (SUEL) and VACUUM (w/o VACOPT_SKIP_LOCKED)
> waiting for the same lock. Once REPACK releases its SUEL, VACUUM gets it and
> processes the table, then REPACK finally gets AccessExclusiveLock (AEL) and
> finishes too.
> One more thing we may prevent from sneaking into that hole is a
> VACUUM. It will not break anything, but will be huge waste of time and
> resources.
I thought about that too, I think we may just add some kind of
CheckTableNotInUse in VACUUM after getting the SUEL.
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 09:11 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-09 09:26 ` Antonin Houska <[email protected]>
2026-04-09 14:06 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-09 09:26 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Mihail Nikalayeu <[email protected]> wrote:
> On Thu, Apr 9, 2026 at 10:43 AM Antonin Houska <[email protected]> wrote:
> > This approach LGTM when it comes to concurrent DDLs. However, consider REPACK
> > holding ShareUpdateExclusiveLock (SUEL) and VACUUM (w/o VACOPT_SKIP_LOCKED)
> > waiting for the same lock. Once REPACK releases its SUEL, VACUUM gets it and
> > processes the table, then REPACK finally gets AccessExclusiveLock (AEL) and
> > finishes too.
>
> > One more thing we may prevent from sneaking into that hole is a
> > VACUUM. It will not break anything, but will be huge waste of time and
> > resources.
>
>
> I thought about that too, I think we may just add some kind of
> CheckTableNotInUse in VACUUM after getting the SUEL.
Sure, it's possible, but IMO the principal question is whether REPACK should
let VACUUM and DDLs error out, or just let them wait.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 09:11 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 09:26 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-09 14:06 ` Mihail Nikalayeu <[email protected]>
2026-04-09 14:26 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-09 14:06 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Amit Kapila <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Thu, Apr 9, 2026 at 11:26 AM Antonin Houska <[email protected]> wrote:
> Sure, it's possible, but IMO the principal question is whether REPACK should
> let VACUUM and DDLs error out, or just let them wait.
One more idea: instead of ERROR in CheckTableNotInUse in case of
in_repack - just release the lock and retry a little later (by that
time, the repack operation will likely have acquired the AEL).
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 09:11 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 09:26 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:06 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-09 14:26 ` Andres Freund <[email protected]>
2026-04-09 14:34 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Andres Freund @ 2026-04-09 14:26 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-09 16:06:17 +0200, Mihail Nikalayeu wrote:
> On Thu, Apr 9, 2026 at 11:26 AM Antonin Houska <[email protected]> wrote:
> > Sure, it's possible, but IMO the principal question is whether REPACK should
> > let VACUUM and DDLs error out, or just let them wait.
>
> One more idea: instead of ERROR in CheckTableNotInUse in case of
> in_repack - just release the lock and retry a little later (by that
> time, the repack operation will likely have acquired the AEL).
I continue to think this approach has no chance of working. Even if it
theoretically could, there are lots of paths to locking relations that do not
go through CheckTableNotInUse(), and we're not going to just route them all
through CheckTableNotInUse().
What CheckTableNotInUse() is for is to prevent DDL from changing the structure
of the table when it is still being referred to. You can't just call
CheckTableNotInUse() from a LOCK TABLE - it would trigger wrong errors *ALL
THE TIME* because a LOCK TABLE does not need to error out just because there's
also a cursor on the table.
And it'd trigger lots of bogus errors. See the first example in
https://postgr.es/m/fpr4nsmyy3mpfrm2mijspr44dgol2cjeke5tyznb4btsznxsgx%40iifdbfe2wl63
S2 would get the lock and then error out due to the proposed check. Even
though there's no need for it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 09:11 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 09:26 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:06 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 14:26 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-09 14:34 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-09 14:34 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-09 10:26:22 -0400, Andres Freund wrote:
> On 2026-04-09 16:06:17 +0200, Mihail Nikalayeu wrote:
> > On Thu, Apr 9, 2026 at 11:26 AM Antonin Houska <[email protected]> wrote:
> > > Sure, it's possible, but IMO the principal question is whether REPACK should
> > > let VACUUM and DDLs error out, or just let them wait.
> >
> > One more idea: instead of ERROR in CheckTableNotInUse in case of
> > in_repack - just release the lock and retry a little later (by that
> > time, the repack operation will likely have acquired the AEL).
>
> I continue to think this approach has no chance of working. Even if it
> theoretically could, there are lots of paths to locking relations that do not
> go through CheckTableNotInUse(), and we're not going to just route them all
> through CheckTableNotInUse().
>
> What CheckTableNotInUse() is for is to prevent DDL from changing the structure
> of the table when it is still being referred to. You can't just call
> CheckTableNotInUse() from a LOCK TABLE - it would trigger wrong errors *ALL
> THE TIME* because a LOCK TABLE does not need to error out just because there's
> also a cursor on the table.
>
>
> And it'd trigger lots of bogus errors. See the first example in
> https://postgr.es/m/fpr4nsmyy3mpfrm2mijspr44dgol2cjeke5tyznb4btsznxsgx%40iifdbfe2wl63
>
> S2 would get the lock and then error out due to the proposed check. Even
> though there's no need for it.
Before you protest that you could just let the non-DDL operation have its
lock, sure, but that's an utterly terrible idea, because it will lead to more
and more work accumulating that then has to happen when the AEL is actually
held.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-09 14:20 ` Andres Freund <[email protected]>
2026-04-10 16:14 ` Re: Adding REPACK [concurrently] Robert Treat <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-09 14:20 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-09 10:43:14 +0200, Antonin Houska wrote:
> What Andres proposed (AFAIU) should help to avoid this problem because
> REPACK's request for AEL would get in front of the VACUUM's request for SUEL
> in the queue.
Note that that already happens today.
This works today (without the error triggering patch):
S1: REPACK starts
S2: LOCK TABLE / VACUUM / ... starts waiting
S1: REPACK tries to get AEL
S1: REPACK's lock requests get reordered in the wait queue to be before S2 and
just gets the lock
S1: REPACK finishes
S2: lock acquisition completes.
That's because we do already have this "jumping the wait queue" logic, which I
had forgotten about.
What does *not* work is this:
S1: REPACK starts
S2: BEGIN; SELECT 1 FROM table LIMIT 1;
S2: LOCK TABLE / VACUUM / ... starts waiting
S1: REPACK tries to get AEL
S1: lock is not granted, can't be reordered to be before S2, because S2 holds
conflicting lock, deadlock detector triggers
S2: lock acquisition completes
But with my proposal to properly teach the deadlock detector about assuming
there's a wait edge for the eventual lock upgrade by S1, the first example
would still work, because the lock upgrade would not be considered a hard
cycle, and the second example would have S2 error out.
> Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it should
> possibly have higher priority than REPACK), but I think this prioritization
> should be implemented in other way than just letting it get in the way of
> REPACK (at the time REPACK is nearly finished).
Yea, it makes no sense to interrupt the long running repack, given that the
new relation will have much less stuff for vacuum to do.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-10 16:14 ` Robert Treat <[email protected]>
2026-04-10 18:56 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-12 14:02 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Robert Treat @ 2026-04-10 16:14 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>
On Thu, Apr 9, 2026 at 10:20 AM Andres Freund <[email protected]> wrote:
> On 2026-04-09 10:43:14 +0200, Antonin Houska wrote:
> > What Andres proposed (AFAIU) should help to avoid this problem because
> > REPACK's request for AEL would get in front of the VACUUM's request for SUEL
> > in the queue.
>
> Note that that already happens today.
>
> This works today (without the error triggering patch):
>
> S1: REPACK starts
> S2: LOCK TABLE / VACUUM / ... starts waiting
> S1: REPACK tries to get AEL
> S1: REPACK's lock requests get reordered in the wait queue to be before S2 and
> just gets the lock
> S1: REPACK finishes
> S2: lock acquisition completes.
>
> That's because we do already have this "jumping the wait queue" logic, which I
> had forgotten about.
>
You know, I was wondering how this wasn't already a problem for
pg_repack/pg_squeeze, and I guess this explains it :-P
>
> What does *not* work is this:
>
> S1: REPACK starts
> S2: BEGIN; SELECT 1 FROM table LIMIT 1;
> S2: LOCK TABLE / VACUUM / ... starts waiting
> S1: REPACK tries to get AEL
> S1: lock is not granted, can't be reordered to be before S2, because S2 holds
> conflicting lock, deadlock detector triggers
> S2: lock acquisition completes
>
> But with my proposal to properly teach the deadlock detector about assuming
> there's a wait edge for the eventual lock upgrade by S1, the first example
> would still work, because the lock upgrade would not be considered a hard
> cycle, and the second example would have S2 error out.
>
In the above S2 will error out if you try to run a VACUUM, but the
point still stands that calling an explicit LOCK or similar could lead
to this issue. In the current repack world, we document the need for
lock escalation at the end of the repacking and caution that doing
things like DDL or explicit LOCKing could cause trouble, so don't do
that. What you're proposing above would be an improvement though,
IMHO.
>
> > Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it should
> > possibly have higher priority than REPACK), but I think this prioritization
> > should be implemented in other way than just letting it get in the way of
> > REPACK (at the time REPACK is nearly finished).
>
> Yea, it makes no sense to interrupt the long running repack, given that the
> new relation will have much less stuff for vacuum to do.
>
We might be talking about 2 different scenarios. In the case where we
are at the point of lock escalation, you would probably want the
repack to get priority over a waiting vacuum, even a failsafe vacuum.
But outside of that scenario, we can't know that the repack is the
better option (and statistically it probably isn't) since a repack
that is actively copying rows might still need to rebuild some large
number of indexes (or just some really expensive index) which could
take significantly longer than a failsafe vacuum would need to ensure
wraparound avoidance. I don't think we'd go as far as saying the
failsafe vacuum should cancel the repack, but I think ideally we'd
like it to not be canceled either, since that would increase
likelihood for dba/monitoring to pick up on the situation, and in the
case that REPACK fails for some reason, the failsafe vacuum could
immediately start working without having to go through any additional
hoops.
Robert Treat
https://xzilla.net
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-10 16:14 ` Re: Adding REPACK [concurrently] Robert Treat <[email protected]>
@ 2026-04-10 18:56 ` Antonin Houska <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-10 18:56 UTC (permalink / raw)
To: Robert Treat <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>
Robert Treat <[email protected]> wrote:
> On Thu, Apr 9, 2026 at 10:20 AM Andres Freund <[email protected]> wrote:
> > On 2026-04-09 10:43:14 +0200, Antonin Houska wrote:
> > > Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it should
> > > possibly have higher priority than REPACK), but I think this prioritization
> > > should be implemented in other way than just letting it get in the way of
> > > REPACK (at the time REPACK is nearly finished).
> >
> > Yea, it makes no sense to interrupt the long running repack, given that the
> > new relation will have much less stuff for vacuum to do.
> >
>
> We might be talking about 2 different scenarios. In the case where we
> are at the point of lock escalation, you would probably want the
> repack to get priority over a waiting vacuum, even a failsafe vacuum.
> But outside of that scenario, we can't know that the repack is the
> better option (and statistically it probably isn't) since a repack
> that is actively copying rows might still need to rebuild some large
> number of indexes (or just some really expensive index) which could
> take significantly longer than a failsafe vacuum would need to ensure
> wraparound avoidance. I don't think we'd go as far as saying the
> failsafe vacuum should cancel the repack, but I think ideally we'd
> like it to not be canceled either, since that would increase
> likelihood for dba/monitoring to pick up on the situation, and in the
> case that REPACK fails for some reason, the failsafe vacuum could
> immediately start working without having to go through any additional
> hoops.
What about just teaching the anti-wraparound VACUUM to print out a WARNING if
it could not lock the table in "reasonable" time? The DBA would then have to
consider if the blocking command should be cancelled, whether it's REPACK or
something else.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-10 16:14 ` Re: Adding REPACK [concurrently] Robert Treat <[email protected]>
@ 2026-04-12 14:02 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-12 14:02 UTC (permalink / raw)
To: Robert Treat <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>
Hi,
On 2026-04-10 12:14:59 -0400, Robert Treat wrote:
> On Thu, Apr 9, 2026 at 10:20 AM Andres Freund <[email protected]> wrote:
> > > Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it should
> > > possibly have higher priority than REPACK), but I think this prioritization
> > > should be implemented in other way than just letting it get in the way of
> > > REPACK (at the time REPACK is nearly finished).
> >
> > Yea, it makes no sense to interrupt the long running repack, given that the
> > new relation will have much less stuff for vacuum to do.
> >
>
> We might be talking about 2 different scenarios. In the case where we
> are at the point of lock escalation, you would probably want the
> repack to get priority over a waiting vacuum, even a failsafe vacuum.
> But outside of that scenario, we can't know that the repack is the
> better option (and statistically it probably isn't) since a repack
> that is actively copying rows might still need to rebuild some large
> number of indexes (or just some really expensive index) which could
> take significantly longer than a failsafe vacuum would need to ensure
> wraparound avoidance.
I don't really understand why you consider REPACK CONCURRENTLY to be so
different from other existing long-running operations?
> I don't think we'd go as far as saying the failsafe vacuum should cancel the
> repack, but I think ideally we'd like it to not be canceled either
I don't think what I suggested could lead to (auto-)vacuum getting cancelled?
Antonin's earlier patch could, but as long as you only do cancellations if
there's an actual deadlock cycle, VACUUM should not get cancelled, since it
won't already hold a lower level lock?
> , since that would increase likelihood for dba/monitoring to pick up on the
> situation, and in the case that REPACK fails for some reason, the failsafe
> vacuum could immediately start working without having to go through any
> additional hoops.
Not sure I buy this (leaving aside the premise doesn't hold, I think). If you
have monitoring for either long running tasks or tables not getting
autovacuumed, it'd pick up on that regardless of whether autovacuum is getting
cancelled or not. What realistic monitoring would pick up on autovacuum just
being stuck waiting for a lock for long, but would not pick up on repack
running forever or age(datfrozenxid) getting a bit long in the tooth?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-12 13:31 ` Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-12 13:31 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Thu, Apr 9, 2026 at 4:20 PM Andres Freund <[email protected]> wrote:
> But with my proposal to properly teach the deadlock detector about
assuming
> there's a wait edge for the eventual lock upgrade by S1, the first example
> would still work, because the lock upgrade would not be considered a hard
> cycle, and the second example would have S2 error out.
Attached patch contains some (maybe naive) POC for similar approach.
It adds a 'deadlock_protected' flag, which changes how the deadlock
detector cancels backends.
Instead of cancelling the backend entered the deadlock detector - it
cancel some another (nearest hard edge) until it is possible to get the
lock (either by
reordering or directly).
Also, I added test cases with the scenarios you mentioned into the repack
spec - to ensure repack is still working while other backend are cancelled.
> > Anti-wraparound (failsafe) VACUUM is a bit different case [1] (i.e. it
should
> > possibly have higher priority than REPACK), but I think this
prioritization
> > should be implemented in other way than just letting it get in the way
of
> > REPACK (at the time REPACK is nearly finished).
>
> Yea, it makes no sense to interrupt the long running repack, given that
the
> new relation will have much less stuff for vacuum to do.
For now I decided to keep anti-wraparound unaffected because the
feature is may be used not only for repack but also for other potential
scenarios.
But yes, maybe it's worth canceling antiwraparound in the repack case.
Also, checking proc flags to detect antuwraparound requires
LWLockConditionalAcquire(ProcArrayLock) - something I don't like in
deadlock detector.
Best regards,
Mikhail.
Attachments:
[application/x-patch] nocfbot-v1-0001-Prefer-canceling-blockers-for-deadlock-protected-.patch (25.7K, 3-nocfbot-v1-0001-Prefer-canceling-blockers-for-deadlock-protected-.patch)
download | inline diff:
From 72414a340b39bc814c4a77b5a6250bae4f85e079 Mon Sep 17 00:00:00 2001
From: Mikhail Nikalayeu <[email protected]>
Date: Sun, 12 Apr 2026 14:24:02 +0200
Subject: [PATCH v1] Prefer canceling blockers for deadlock-protected repack
waits
Teach the lock manager to treat selected waits as deadlock-protected, so
REPACK CONCURRENTLY can preserve the work done before waiting for the
final AccessExclusiveLock set.
When deadlock_protected is set and a hard deadlock is found, cancel a
blocking lock wait and let the protected backend continue waiting
instead of making it the deadlock victim. Keep anti-wraparound
autovacuum workers ineligible as cancellation targets, using a
conditional ProcArrayLock check for the wraparound flag so deadlock
detection does not block on LWLocks.
Set this protection around repack's final lock acquisition and clear it
through the normal wait/transaction cleanup paths.
---
src/backend/commands/repack.c | 8 ++
src/backend/storage/ipc/procarray.c | 3 +
src/backend/storage/lmgr/deadlock.c | 103 +++++++++++++-
src/backend/storage/lmgr/lock.c | 18 ++-
src/backend/storage/lmgr/proc.c | 63 +++++++--
src/include/storage/lock.h | 6 +-
src/include/storage/proc.h | 9 +-
.../injection_points/expected/repack.out | 130 +++++++++++++++++-
.../injection_points/specs/repack.spec | 80 +++++++++++
9 files changed, 399 insertions(+), 21 deletions(-)
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 58e3867246f..cee2aa49c06 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -3055,7 +3055,15 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
/*
* Acquire AccessExclusiveLock on the table, its TOAST relation (if there
* is one), all its indexes, so that we can swap the files.
+ *
+ * Set deadlock_protected so that if a deadlock occurs while we are
+ * waiting for these locks, the deadlock detector cancels the blocking
+ * proc's lock wait instead of ours. At this point we have done the
+ * expensive data copy and index rebuild - aborting now would waste all
+ * that work.
*/
+ deadlock_protected = true;
+
LockRelationOid(old_table_oid, AccessExclusiveLock);
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9299bcebbda..1dbbae54017 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -714,6 +714,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
LWLockRelease(ProcArrayLock);
}
}
+
+ /* Clear backend-local deadlock protection flag at end of transaction. */
+ deadlock_protected = false;
}
/*
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index b8962d875b6..7b53e2fb19a 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -125,6 +125,15 @@ static int maxPossibleConstraints;
static DEADLOCK_INFO *deadlockDetails;
static int nDeadlockDetails;
+/*
+ * Per-edge parallel array to deadlockDetails[]: the waiting proc whose wait
+ * forms the edge at the corresponding index. Unlike deadlockDetails[], these
+ * pointers reference live PGPROCs and are only valid while all partition
+ * locks are held (i.e., during DeadLockCheck). Used by deadlock_protected
+ * resolution in DeadLockCheck to pick a cancellation victim.
+ */
+static PGPROC **deadlockProcs;
+
/* PGPROC pointer of any blocking autovacuum worker found */
static PGPROC *blocking_autovacuum_proc = NULL;
@@ -148,11 +157,12 @@ InitDeadLockChecking(void)
oldcxt = MemoryContextSwitchTo(TopMemoryContext);
/*
- * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
- * deadlockDetails[].
+ * FindLockCycle needs at most MaxBackends entries in visitedProcs[],
+ * deadlockDetails[], and deadlockProcs[].
*/
visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+ deadlockProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
/*
* TopoSort needs to consider at most MaxBackends wait-queue entries, and
@@ -219,6 +229,9 @@ InitDeadLockChecking(void)
DeadLockState
DeadLockCheck(PGPROC *proc)
{
+ bool victims_canceled = false;
+
+retry:
/* Initialize to "no constraints" */
nCurConstraints = 0;
nPossibleConstraints = 0;
@@ -242,6 +255,76 @@ DeadLockCheck(PGPROC *proc)
if (!FindLockCycle(proc, possibleConstraints, &nSoftEdges))
elog(FATAL, "deadlock seems to have disappeared");
+ /*
+ * If deadlock_protected is set, try to resolve the deadlock by
+ * canceling a blocking proc's lock wait, then restarting the whole
+ * deadlock check. We re-run DeadLockCheckRecurse (not just
+ * FindLockCycle) after each cancellation so that any remaining cycle
+ * also gets a chance to be resolved by the ordinary soft-deadlock
+ * queue-reordering path.
+ *
+ * The victim is chosen by walking deadlockProcs[] which FindLockCycle
+ * just populated with one live PGPROC pointer per edge in the cycle.
+ * Index 0 is always MyProc (we started the search from it), so we
+ * start at index 1, which is the direct blocker of MyProc.
+ *
+ * We skip any proc running an anti-wraparound autovacuum: canceling that
+ * would risk XID wraparound, which is strictly worse than losing the
+ * work the deadlock_protected caller is trying to preserve. XXX: is it true?
+ *
+ * If no acceptable victim exists, we fall through to DS_HARD_DEADLOCK
+ * and the protected proc cancels itself as usual.
+ */
+ if (deadlock_protected)
+ {
+ PGPROC *victim = NULL;
+
+ for (int i = 1; i < nDeadlockDetails; i++)
+ {
+ PGPROC *w = deadlockProcs[i];
+
+ if (w == MyProc)
+ continue;
+
+ /*
+ * PROC_IS_AUTOVACUUM is safe to check locklessly because it is
+ * set at process start and never reset. For autovac workers,
+ * avoid waiting for ProcArrayLock here; if we can acquire it
+ * conditionally, check the mirrored wraparound flag to avoid
+ * canceling an emergency vacuum. If not, skip this candidate
+ * conservatively and keep looking for another victim.
+ */
+ if (w->statusFlags & PROC_IS_AUTOVACUUM)
+ {
+ uint8 statusFlags;
+
+ if (!LWLockConditionalAcquire(ProcArrayLock, LW_SHARED))
+ continue;
+
+ statusFlags = ProcGlobal->statusFlags[w->pgxactoff];
+ LWLockRelease(ProcArrayLock);
+
+ if (statusFlags & PROC_VACUUM_FOR_WRAPAROUND)
+ continue;
+ }
+
+ Assert(w->waitLock != NULL);
+ Assert(!dlist_node_is_detached(&w->waitLink));
+ victim = w;
+ break;
+ }
+
+ if (victim != NULL)
+ {
+ RemoveFromWaitQueue(victim,
+ LockTagHashCode(&(victim->waitLock->tag)),
+ PROC_WAIT_STATUS_DEADLOCK_CANCEL);
+ SetLatch(&victim->procLatch);
+ victims_canceled = true;
+ goto retry;
+ }
+ }
+
return DS_HARD_DEADLOCK; /* cannot find a non-deadlocked state */
}
@@ -272,8 +355,16 @@ DeadLockCheck(PGPROC *proc)
ProcLockWakeup(GetLocksMethodTable(lock), lock);
}
- /* Return code tells caller if we had to escape a deadlock or not */
- if (nWaitOrders > 0)
+ /*
+ * Return code tells caller if we had to escape a deadlock or not.
+ *
+ * If we canceled any waiter because deadlock_protected was set, report
+ * DS_HARD_DEADLOCK_CANCELED regardless of whether the remainder was
+ * resolved by queue rearrangement or needed nothing further.
+ */
+ if (victims_canceled)
+ return DS_HARD_DEADLOCK_CANCELED;
+ else if (nWaitOrders > 0)
return DS_SOFT_DEADLOCK;
else if (blocking_autovacuum_proc != NULL)
return DS_BLOCKED_BY_AUTOVACUUM;
@@ -590,6 +681,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
return true;
}
@@ -679,6 +771,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
/*
* Add this edge to the list of soft edges in the cycle
@@ -753,6 +846,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
/*
* Add this edge to the list of soft edges in the cycle
@@ -1159,4 +1253,5 @@ RememberSimpleDeadLock(PGPROC *proc1,
info->lockmode = proc2->waitLockMode;
info->pid = proc2->pid;
nDeadlockDetails = 2;
+ deadlockProcs[0] = deadlockProcs[1] = NULL;
}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index c221fe96889..1a268f05965 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -1245,6 +1245,14 @@ LockAcquireExtended(const LOCKTAG *locktag,
DeadLockReport();
/* DeadLockReport() will not return */
}
+ else if (waitResult == PROC_WAIT_STATUS_DEADLOCK_CANCEL)
+ {
+ Assert(!dontWait);
+ ereport(ERROR,
+ (errcode(ERRCODE_T_R_DEADLOCK_DETECTED),
+ errmsg("deadlock detected"),
+ errdetail("This backend's lock wait was canceled to resolve a deadlock with a protected process.")));
+ }
}
else
LWLockRelease(partitionLock);
@@ -2043,8 +2051,8 @@ waitonlock_error_callback(void *arg)
/*
* Remove a proc from the wait-queue it is on (caller must know it is on one).
- * This is only used when the proc has failed to get the lock, so we set its
- * waitStatus to PROC_WAIT_STATUS_ERROR.
+ * This is only used when the proc has failed to get the lock, so caller must
+ * specify the failure waitStatus to store.
*
* Appropriate partition lock must be held by caller. Also, caller is
* responsible for signaling the proc if needed.
@@ -2052,7 +2060,7 @@ waitonlock_error_callback(void *arg)
* NB: this does not clean up any locallock object that may exist for the lock.
*/
void
-RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
+RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode, ProcWaitStatus waitStatus)
{
LOCK *waitLock = proc->waitLock;
PROCLOCK *proclock = proc->waitProcLock;
@@ -2065,6 +2073,8 @@ RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
Assert(waitLock);
Assert(!dclist_is_empty(&waitLock->waitProcs));
Assert(0 < lockmethodid && lockmethodid < lengthof(LockMethods));
+ Assert(waitStatus == PROC_WAIT_STATUS_ERROR ||
+ waitStatus == PROC_WAIT_STATUS_DEADLOCK_CANCEL);
/* Remove proc from lock's wait queue */
dclist_delete_from_thoroughly(&waitLock->waitProcs, &proc->waitLink);
@@ -2082,7 +2092,7 @@ RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
/* Clean up the proc's own state, and pass it the ok/fail signal */
proc->waitLock = NULL;
proc->waitProcLock = NULL;
- proc->waitStatus = PROC_WAIT_STATUS_ERROR;
+ proc->waitStatus = waitStatus;
/*
* Delete the proclock immediately if it represents no already-held locks.
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1ac25068d62..8afc247bebe 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -92,6 +92,9 @@ static size_t FastPathLockArrayShmemSize;
/* Is a deadlock check pending? */
static volatile sig_atomic_t got_deadlock_timeout;
+/* See proc.h */
+bool deadlock_protected = false;
+
static void RemoveProcFromArray(int code, Datum arg);
static void ProcKill(int code, Datum arg);
static void AuxiliaryProcKill(int code, Datum arg);
@@ -825,6 +828,8 @@ LockErrorCleanup(void)
AbortStrongLockAcquire();
+ deadlock_protected = false;
+
/* Nothing to do if we weren't waiting for a lock */
lockAwaited = GetAwaitedLock();
if (lockAwaited == NULL)
@@ -854,7 +859,7 @@ LockErrorCleanup(void)
if (!dlist_node_is_detached(&MyProc->waitLink))
{
/* We could not have been granted the lock yet */
- RemoveFromWaitQueue(MyProc, lockAwaited->hashcode);
+ RemoveFromWaitQueue(MyProc, lockAwaited->hashcode, PROC_WAIT_STATUS_ERROR);
}
else
{
@@ -1266,8 +1271,16 @@ JoinWaitQueue(LOCALLOCK *locallock, LockMethod lockMethodTable, bool dontWait)
/*
* If we detected deadlock, give up without waiting. This must agree with
* CheckDeadLock's recovery code.
+ *
+ * However, if deadlock_protected is set, skip the early exit and proceed
+ * to ProcSleep so the full deadlock detector can handle it (by making a
+ * blocking proc the victim instead of us).
+ *
+ * XXX: when we bypass the early-exit here, we lose the ability to react to
+ * the deadlock immediately: ProcSleep will only run the full detector
+ * after deadlock_timeout elapses. Should we fix that?
*/
- if (early_deadlock)
+ if (early_deadlock && !deadlock_protected)
return PROC_WAIT_STATUS_ERROR;
/*
@@ -1308,8 +1321,10 @@ JoinWaitQueue(LOCALLOCK *locallock, LockMethod lockMethodTable, bool dontWait)
*
* Result is one of the following:
*
- * PROC_WAIT_STATUS_OK - lock was granted
- * PROC_WAIT_STATUS_ERROR - a deadlock was detected
+ * PROC_WAIT_STATUS_OK - lock was granted
+ * PROC_WAIT_STATUS_ERROR - a deadlock was detected
+ * PROC_WAIT_STATUS_DEADLOCK_CANCEL - this backend was chosen as the
+ * victim of a protected deadlock
*/
ProcWaitStatus
ProcSleep(LOCALLOCK *locallock)
@@ -1345,8 +1360,8 @@ ProcSleep(LOCALLOCK *locallock)
/*
* Set timer so we can wake up after awhile and check for a deadlock. If a
* deadlock is detected, the handler sets MyProc->waitStatus =
- * PROC_WAIT_STATUS_ERROR, allowing us to know that we must report failure
- * rather than success.
+ * one of the failure wait statuses, allowing us to know that we must
+ * report failure rather than success.
*
* By delaying the check until we've waited for a bit, we can avoid
* running the rather expensive deadlock-check code in most cases.
@@ -1560,6 +1575,17 @@ ProcSleep(LOCALLOCK *locallock)
allow_autovacuum_cancel = false;
}
+ /*
+ * If the deadlock detector resolved a hard deadlock by canceling a
+ * blocking proc's lock wait (because we have deadlock_protected set),
+ * restart the deadlock timeout and continue waiting for the lock.
+ */
+ if (deadlock_state == DS_HARD_DEADLOCK_CANCELED)
+ {
+ got_deadlock_timeout = false;
+ enable_timeout_after(DEADLOCK_TIMEOUT, DeadlockTimeout);
+ }
+
/*
* If awoken after the deadlock check interrupt has run, increment the
* lock statistics counters and if log_lock_waits is on, then report
@@ -1627,6 +1653,13 @@ ProcSleep(LOCALLOCK *locallock)
"Processes holding the lock: %s. Wait queue: %s.",
lockHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data))));
}
+ else if (deadlock_state == DS_HARD_DEADLOCK_CANCELED)
+ ereport(LOG,
+ (errmsg("process %d resolved deadlock for %s on %s by canceling a blocking lock wait after %ld.%03d ms",
+ MyProcPid, modename, buf.data, msecs, usecs),
+ (errdetail_log_plural("Process holding the lock: %s. Wait queue: %s.",
+ "Processes holding the lock: %s. Wait queue: %s.",
+ lockHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data))));
if (myWaitStatus == PROC_WAIT_STATUS_WAITING)
{
@@ -1657,13 +1690,21 @@ ProcSleep(LOCALLOCK *locallock)
ereport(LOG,
(errmsg("process %d acquired %s on %s after %ld.%03d ms",
MyProcPid, modename, buf.data, msecs, usecs)));
+ else if (myWaitStatus == PROC_WAIT_STATUS_DEADLOCK_CANCEL)
+ ereport(LOG,
+ (errmsg("process %d was chosen as deadlock victim for %s on %s after %ld.%03d ms",
+ MyProcPid, modename, buf.data, msecs, usecs),
+ (errdetail_log_plural("Process holding the lock: %s. Wait queue: %s.",
+ "Processes holding the lock: %s. Wait queue: %s.",
+ lockHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data))));
else
{
Assert(myWaitStatus == PROC_WAIT_STATUS_ERROR);
/*
- * Currently, the deadlock checker always kicks its own
- * process, which means that we'll only see
+ * A plain PROC_WAIT_STATUS_ERROR means the deadlock
+ * checker kicked our own process, which means that we'll
+ * only see
* PROC_WAIT_STATUS_ERROR when deadlock_state ==
* DS_HARD_DEADLOCK, and there's no need to print
* redundant messages. But for completeness and
@@ -1870,14 +1911,16 @@ CheckDeadLock(void)
* Get this process out of wait state. (Note: we could do this more
* efficiently by relying on lockAwaited, but use this coding to
* preserve the flexibility to kill some other transaction than the
- * one detecting the deadlock.)
+ * one detecting the deadlock. (DS_HARD_DEADLOCK_CANCELED above
+ * already exercises that flexibility: when deadlock_protected is set,
+ * we cancel a *blocking* proc's lock wait instead of our own.)
*
* RemoveFromWaitQueue sets MyProc->waitStatus to
* PROC_WAIT_STATUS_ERROR, so ProcSleep will report an error after we
* return.
*/
Assert(MyProc->waitLock != NULL);
- RemoveFromWaitQueue(MyProc, LockTagHashCode(&(MyProc->waitLock->tag)));
+ RemoveFromWaitQueue(MyProc, LockTagHashCode(&(MyProc->waitLock->tag)), PROC_WAIT_STATUS_ERROR);
/*
* We're done here. Transaction abort caused by the error that
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index ee3cb1dc203..5e13de529c5 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -29,6 +29,7 @@
/* struct PGPROC is declared in proc.h, but must forward-reference it */
typedef struct PGPROC PGPROC;
+typedef enum ProcWaitStatus ProcWaitStatus;
/* GUC variables */
extern PGDLLIMPORT int max_locks_per_xact;
@@ -342,6 +343,9 @@ typedef enum
DS_NO_DEADLOCK, /* no deadlock detected */
DS_SOFT_DEADLOCK, /* deadlock avoided by queue rearrangement */
DS_HARD_DEADLOCK, /* deadlock, no way out but ERROR */
+ DS_HARD_DEADLOCK_CANCELED, /* deadlock resolved by canceling a blocking
+ * proc's lock wait (when
+ * deadlock_protected is set) */
DS_BLOCKED_BY_AUTOVACUUM, /* no deadlock; queue blocked by autovacuum
* worker */
} DeadLockState;
@@ -418,7 +422,7 @@ extern void GrantAwaitedLock(void);
extern LOCALLOCK *GetAwaitedLock(void);
extern void ResetAwaitedLock(void);
-extern void RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode);
+extern void RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode, ProcWaitStatus waitStatus);
extern LockData *GetLockStatusData(void);
extern BlockedProcsData *GetBlockerStatusData(int blocked_pid);
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 3e1d1fad5f9..ef02639834e 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -146,11 +146,12 @@ extern PGDLLIMPORT int FastPathLockGroupsPerBackend;
#define DELAY_CHKPT_COMPLETE (1<<1)
#define DELAY_CHKPT_IN_COMMIT (DELAY_CHKPT_START | 1<<2)
-typedef enum
+typedef enum ProcWaitStatus
{
PROC_WAIT_STATUS_OK,
PROC_WAIT_STATUS_WAITING,
PROC_WAIT_STATUS_ERROR,
+ PROC_WAIT_STATUS_DEADLOCK_CANCEL,
} ProcWaitStatus;
/*
@@ -543,6 +544,12 @@ extern PGDLLIMPORT int TransactionTimeout;
extern PGDLLIMPORT int IdleSessionTimeout;
extern PGDLLIMPORT bool log_lock_waits;
+/*
+ * Backend-local flag: when true, the deadlock detector will make a blocking
+ * proc the victim instead of this backend.
+ */
+extern PGDLLIMPORT bool deadlock_protected;
+
#ifdef EXEC_BACKEND
extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
#endif
diff --git a/src/test/modules/injection_points/expected/repack.out b/src/test/modules/injection_points/expected/repack.out
index b575e9052ee..ae706125360 100644
--- a/src/test/modules/injection_points/expected/repack.out
+++ b/src/test/modules/injection_points/expected/repack.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: wait_before_lock change_existing change_new change_subxact1 change_subxact2 check2 wakeup_before_lock check1
injection_points_attach
@@ -111,3 +111,131 @@ injection_points_detach
(1 row)
+
+starting permutation: wait_before_lock begin_txn lock_table s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_txn:
+ BEGIN;
+
+step lock_table:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step lock_table: <... completed>
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s1_timeout_fast s2_timeout_slow wait_before_lock begin_and_read lock_table s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s1_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step s2_timeout_slow:
+ SET deadlock_timeout = '10s';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step lock_table: <... completed>
+ERROR: deadlock detected
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s1_timeout_slow s2_timeout_fast wait_before_lock begin_and_read lock_table s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s1_timeout_slow:
+ SET deadlock_timeout = '10s';
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step lock_table: <... completed>
+ERROR: deadlock detected
+step wait_before_lock: <... completed>
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/specs/repack.spec b/src/test/modules/injection_points/specs/repack.spec
index d727a9b056b..f95f6782355 100644
--- a/src/test/modules/injection_points/specs/repack.spec
+++ b/src/test/modules/injection_points/specs/repack.spec
@@ -28,6 +28,14 @@ setup
SELECT injection_points_set_local();
SELECT injection_points_attach('repack-concurrently-before-lock', 'wait');
}
+step s1_timeout_fast
+{
+ SET deadlock_timeout = '10ms';
+}
+step s1_timeout_slow
+{
+ SET deadlock_timeout = '10s';
+}
# Perform the initial load and wait for s2 to do some data changes.
step wait_before_lock
{
@@ -61,6 +69,14 @@ teardown
}
session s2
+step s2_timeout_fast
+{
+ SET deadlock_timeout = '10ms';
+}
+step s2_timeout_slow
+{
+ SET deadlock_timeout = '10s';
+}
# Change the existing data. UPDATE changes both key and non-key columns. Also
# update one row twice to test whether tuple version generated by this session
# can be found.
@@ -128,6 +144,30 @@ step wakeup_before_lock
{
SELECT injection_points_wakeup('repack-concurrently-before-lock');
}
+# Steps used in lock contention tests.
+step begin_txn
+{
+ BEGIN;
+}
+step begin_and_read
+{
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+}
+step lock_table
+{
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+}
+step end_txn
+{
+ COMMIT;
+}
+
+session s3
+step s3_wakeup
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
# Test if data changes introduced while one session is performing REPACK
# CONCURRENTLY find their way into the table.
@@ -140,3 +180,43 @@ permutation
check2
wakeup_before_lock
check1
+
+# Test the soft-deadlock path of REPACK CONCURRENTLY's deadlock_protected
+# behavior. s2 is queued ahead of s1 (REPACK) waiting for AccessExclusiveLock
+# but holds no lock on the table itself, so the cycle has only soft edges and
+# the deadlock detector resolves it by reordering the wait queue. Both
+# sessions complete successfully.
+permutation
+ wait_before_lock
+ begin_txn
+ lock_table
+ s3_wakeup
+ end_txn
+
+# Test the hard-deadlock path of REPACK CONCURRENTLY's deadlock_protected
+# behavior when REPACK's deadlock detector runs first. s2 holds
+# AccessShareLock (from the SELECT) and then requests AccessExclusiveLock,
+# creating a hard cycle with s1 (REPACK) that cannot be resolved by queue
+# reordering. Because s1 has a much shorter deadlock_timeout and sets
+# deadlock_protected before acquiring AccessExclusiveLock, its deadlock check
+# runs first, cancels s2's lock wait, and lets REPACK complete.
+permutation
+ s1_timeout_fast
+ s2_timeout_slow
+ wait_before_lock
+ begin_and_read
+ lock_table(wait_before_lock)
+ s3_wakeup
+ end_txn
+
+# Same hard-deadlock setup, but with s2's deadlock detector forced to run
+# first. In this case s2 follows the ordinary hard-deadlock path and cancels
+# itself before REPACK's protected deadlock detector has a chance to act.
+permutation
+ s1_timeout_slow
+ s2_timeout_fast
+ wait_before_lock(lock_table)
+ begin_and_read
+ lock_table
+ s3_wakeup
+ end_txn
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-12 14:05 ` Andres Freund <[email protected]>
2026-04-12 14:58 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-12 14:05 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-12 15:31:20 +0200, Mihail Nikalayeu wrote:
> On Thu, Apr 9, 2026 at 4:20 PM Andres Freund <[email protected]> wrote:
> > But with my proposal to properly teach the deadlock detector about
> assuming
> > there's a wait edge for the eventual lock upgrade by S1, the first example
> > would still work, because the lock upgrade would not be considered a hard
> > cycle, and the second example would have S2 error out.
>
> Attached patch contains some (maybe naive) POC for similar approach.
> It adds a 'deadlock_protected' flag, which changes how the deadlock
> detector cancels backends.
>
> Instead of cancelling the backend entered the deadlock detector - it
> cancel some another (nearest hard edge) until it is possible to get the
> lock (either by
> reordering or directly).
I don't think that's as good. The problem is that that way you're only
detecting the deadlocks once they have materialized (i.e. once repack actually
does the lock upgrade), rather than cancelling when we know that the problem
starts. Having sessions pointlessly blocked for many hours is bad.
> Also, I added test cases with the scenarios you mentioned into the repack
> spec - to ensure repack is still working while other backend are cancelled.
I think we should perhaps commit spec tests for these (I've not yet reviewed
them, but in principle), even before we fix the problem. It's good to document
the current behavior and have a comment that the wrongly cancelled case should
not trigger an error.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-12 14:58 ` Mihail Nikalayeu <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-12 14:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi!
On Sun, Apr 12, 2026 at 4:05 PM Andres Freund <[email protected]> wrote:
> I don't think that's as good. The problem is that that way you're only
> detecting the deadlocks once they have materialized (i.e. once repack actually
> does the lock upgrade), rather than cancelling when we know that the problem
> starts. Having sessions pointlessly blocked for many hours is bad.
O, I think I understand you now.
You propose to somehow mark SUEL as "going to become AEL, so, the
deadlock detector should treat it as such".
In that case LOCK TABLE should get some kind of "future deadlock detected".
Yep, that feels much better. I'll try to check that approach tomorrow.
Best regards,
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-14 13:37 ` Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-14 13:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Andres Freund <[email protected]> wrote:
> On 2026-04-12 15:31:20 +0200, Mihail Nikalayeu wrote:
> > Instead of cancelling the backend entered the deadlock detector - it
> > cancel some another (nearest hard edge) until it is possible to get the
> > lock (either by
> > reordering or directly).
>
> I don't think that's as good. The problem is that that way you're only
> detecting the deadlocks once they have materialized (i.e. once repack actually
> does the lock upgrade), rather than cancelling when we know that the problem
> starts.
This is my hack that tries to do that.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-14 13:58 ` Andres Freund <[email protected]>
2026-04-14 16:55 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-14 13:58 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-14 15:37:56 +0200, Antonin Houska wrote:
> Andres Freund <[email protected]> wrote:
>
> > On 2026-04-12 15:31:20 +0200, Mihail Nikalayeu wrote:
> > > Instead of cancelling the backend entered the deadlock detector - it
> > > cancel some another (nearest hard edge) until it is possible to get the
> > > lock (either by
> > > reordering or directly).
> >
> > I don't think that's as good. The problem is that that way you're only
> > detecting the deadlocks once they have materialized (i.e. once repack actually
> > does the lock upgrade), rather than cancelling when we know that the problem
> > starts.
>
> This is my hack that tries to do that.
I still think this needs to be in the deadlock detector. The lock cycle just
needs to be a bit more complicated for a hack in JoinWaitQueue not to work.
There's no guarantee that the wait that triggers the deadlock is actually on
the relation being repacked.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-14 16:55 ` Mihail Nikalayeu <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-14 16:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Antonin Houska <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Tue, Apr 14, 2026 at 3:58 PM Andres Freund <[email protected]> wrote:
> I still think this needs to be in the deadlock detector. The lock cycle just
> needs to be a bit more complicated for a hack in JoinWaitQueue not to work.
> There's no guarantee that the wait that triggers the deadlock is actually on
> the relation being repacked.
I have started prototyping a way to declare a "future" lock which the
deadlock detector treats as a hard edge.
But I currently stuck on issues related to the fact that SUE doesn't
force weak locks (fast-path) to go through
FastPathTransferRelationLocks, so the deadlock detector can't handle
the case when another backend tries to execute LOCK TABLE repack_test
IN SHARE UPDATE EXCLUSIVE MODE;
Also, VACUUM takes the same lock.
I'm not sure how to deal with this in a non-hacky way. One option is
to force SUE to transfer locks if relation it is trying to lock
relation which marked with "future lock". But I am not sure it is good
enough or covers all tricky cases (multiple backends in the loop).
Best regards,
Mikhail.
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-15 14:50 ` Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 15:54 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
1 sibling, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-15 14:50 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Andres Freund <[email protected]> wrote:
> On 2026-04-14 15:37:56 +0200, Antonin Houska wrote:
> > Andres Freund <[email protected]> wrote:
> >
> > > On 2026-04-12 15:31:20 +0200, Mihail Nikalayeu wrote:
> > > > Instead of cancelling the backend entered the deadlock detector - it
> > > > cancel some another (nearest hard edge) until it is possible to get the
> > > > lock (either by
> > > > reordering or directly).
> > >
> > > I don't think that's as good. The problem is that that way you're only
> > > detecting the deadlocks once they have materialized (i.e. once repack actually
> > > does the lock upgrade), rather than cancelling when we know that the problem
> > > starts.
> >
> > This is my hack that tries to do that.
>
> I still think this needs to be in the deadlock detector. The lock cycle just
> needs to be a bit more complicated for a hack in JoinWaitQueue not to work.
> There's no guarantee that the wait that triggers the deadlock is actually on
> the relation being repacked.
ok, I see.
I thought of a "hypothetical graph", which would include the to-be-granted
lock, but the major issue is that it will not work correctly without the
locking the LMGR's LW locks we do in CheckDeadLock():
for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);
And obviously, doing this each time we want to insert a lock into the queue
would be bad for performance. It's even mentioned in the storage/lmgr/README
that the current approach is optimistic, so I think that major rework would be
needed if we wanted to entirely avoid waiting that leads to deadlock.
The approach proposed by Mihail [1] seems the least problematic to me, and
something like that occurred to me when I thought about the problem the first
time. However, when we wake up the other processes in order to run the
deadlock detection, they should do that immediately. I've got no good idea
about implementation at the moment, since latch can be set for unrelated
reasons. (Besides that, I have some more questions about this patch, which I
can post separately.)
[1] https://www.postgresql.org/message-id/CADzfLwURKVNQ%2B%2BDpi7bjoGfj-8pchDQEVex3eWBx0NCYn6TbDQ%40mail...
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-15 14:59 ` Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Andres Freund @ 2026-04-15 14:59 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-15 16:50:11 +0200, Antonin Houska wrote:
> I thought of a "hypothetical graph", which would include the to-be-granted
> lock, but the major issue is that it will not work correctly without the
> locking the LMGR's LW locks we do in CheckDeadLock():
>
> for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
> LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);
>
> And obviously, doing this each time we want to insert a lock into the queue
> would be bad for performance.
Hence my suggestion to do this as part of the deadlock check. Then we don't do
this unnecessary work outside of the case where we actually need it.
That does need to deal with the case of the deadlock check running first in
the backend doing repack, but that's not that hard - I think it'd be good
enough to set its deadlock timeout temporarily to a higher value. The backend
*should* still run the deadlock detector, because it could probably still get
into a deadlock (e.g. due to a pg_class access or something).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
@ 2026-04-15 18:28 ` Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-15 18:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Andres Freund <[email protected]> wrote:
> On 2026-04-15 16:50:11 +0200, Antonin Houska wrote:
> > I thought of a "hypothetical graph", which would include the to-be-granted
> > lock, but the major issue is that it will not work correctly without the
> > locking the LMGR's LW locks we do in CheckDeadLock():
> >
> > for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
> > LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);
> >
> > And obviously, doing this each time we want to insert a lock into the queue
> > would be bad for performance.
>
> Hence my suggestion to do this as part of the deadlock check. Then we don't do
> this unnecessary work outside of the case where we actually need it.
> That does need to deal with the case of the deadlock check running first in
> the backend doing repack, but that's not that hard - I think it'd be good
> enough to set its deadlock timeout temporarily to a higher value. The backend
> *should* still run the deadlock detector, because it could probably still get
> into a deadlock (e.g. due to a pg_class access or something).
Yes, the question is when we should run that check. I thought that it should
happen during each lock acquisition, and that made me worried about
performance. AFAIU you suggest REPACK to do something like:
1. Acquire ShareUpdateExclusiveLock
2. Perform the "enhanced check" to see if a future request for
AccessExclusiveLock can trigger a deadlock.
3. Do major part of the work (copy the table, build indexes, ...)
4. Request AccessExclusive lock.
5. Finish the work (process the remaining concurrent changes and swap the
table files).
This makes me concerned that if another session does
BEGIN;
TABLE t;
between steps 2 and 4, and something like
-- Get AccessExclusiveLock
ALTER TABLE t ADD COLUMN j int;
just after step 4, the two session will probably up in a deadlock anyway. In
other words, even if REPACK does the check early, it does not prevent other
sessions from getting in the way.
Maybe I'm still missing something.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-16 11:18 ` Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-16 11:18 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Wed, Apr 15, 2026 at 8:28 PM Antonin Houska <[email protected]> wrote:
> just after step 4, the two session will probably up in a deadlock anyway. In
> other words, even if REPACK does the check early, it does not prevent other
> sessions from getting in the way.
>
> Maybe I'm still missing something.
I was trying to solve that by using another approach: the ability to
define a "future lock" [0]. It is declared even before taking the
actual lock, so, no race is possible.
That approach works correctly except one case -
ShareUpdateExclusiveLock from another backend, I described it here [1]
a little bit.
For now, I don't know how to solve it without a performance downgrade.
[0]: https://github.com/michail-nikolaev/postgres/commit/ba0f4247dad3d96b8282cd18056b7776cd69317c
[1]: https://www.postgresql.org/message-id/flat/CADzfLwU8Qw6LXFHO7Tbjc-O7o%2BtM26jdnOJBWqYLu61rf7bO%2Bg%4...
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-17 02:01 ` Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-17 02:01 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
I think I got working POC for deadlock-detector enhancements for
REPACK (and potentially other).
Each PGPROC gets one FutureWaitLock slot: a (locktag, mode) pair,
which is empty when locktag_lockmethodid == 0.
A backend calls LockDeclareFutureWait() to publish its intent. Only
REPACK (CONCURRENTLY) uses it today, declaring its future
AccessExclusiveLock.
Only the owning backend writes its slot; it does so under the
partition lock covering the declared tag. Remote readers (the deadlock
detector) must hold that partition lock to observe it.
The detector holds all partition locks during its graph walk, so it
can read any slot safely. The slot clears in LockAcquireExtended() at
the exact point the backend attaches its waitLink for the same tag and
mode.
It is also cleared on abort, backend exit, and end of transaction.
FindLockCycleRecurseFuture() treats the declared future lock as a hard
edge to (a) current holders whose mode conflicts and (b)
already-queued waiters that JoinWaitQueue() would place ahead of the
future request.
Weak relation locks are kept in per-backend fast-path arrays and
remain invisible to the detector's graph walk.
Normally LockAcquireExtended() migrates them to the main lock table
when a conflicting strong lock is requested.
This action is gated by FastPathStrongRelationLocks->count to prevent
new fast-path lock processing.
A future-wait declaration does not trigger that migration due to
performance considerations.
Instead CheckDeadLock() now snapshots currently declared
fast-path-incompatible future waits and increments
FastPathStrongRelationLocks->count for those tags (blocking new
fast-path acquisitions).
It then calls FastPathTransferRelationLocks() for each such tag to
move existing holders into the main table, making everything visible
to the deadlock walker.
Once the deadlock checker finishes, FastPathStrongRelationLocks->count
is decremented enabling fast-path processing again. Performance
degradation occurs only during deadlock processing and only if REPACK
is active.
If a deadlock contains a "future" edge it is reported as "future
deadlock detected".
Repack first declares future AE and only then proceeds to actually get SUE.
Added test cases for different scenarios, including SUE from another
backend and a 3-backend deadlock.
Attachments:
[application/octet-stream] nocfbot-v2-0001-Detect-deadlocks-involving-declared-future-lock-r.patch (46.1K, 2-nocfbot-v2-0001-Detect-deadlocks-involving-declared-future-lock-r.patch)
download | inline diff:
From 7e052a5f3aafa3a9ca1d7fc866c597001251bcc2 Mon Sep 17 00:00:00 2001
From: Mikhail Nikalayeu <[email protected]>
Date: Fri, 17 Apr 2026 02:09:40 +0200
Subject: [PATCH v2] Detect deadlocks involving declared future lock requests
Introduce a mechanism for a backend to declare its intent to acquire a
lock at a future point, before actually calling LockAcquire(). The
deadlock detector treats each declaration as a hard waits-for edge from
the declarer to current holders whose mode conflicts, and to already-
queued waiters whose requests would be ordered before the future
request. This makes it possible to detect cycles that the existing
detector cannot see, because the declarer has not yet queued for its
stronger lock.
The motivating caller is REPACK (CONCURRENTLY), which holds a
ShareUpdateExclusiveLock throughout and later needs AccessExclusiveLock
to swap relfilenodes. A concurrent backend that takes an intermediate
lock and waits for REPACK to finish can form a cycle that today is
invisible until REPACK finally requests its strong lock. Declaring the
future AccessExclusiveLock from RangeVarCallbackForRepack surfaces the
edge immediately after the initial lookup.
Implementation notes:
* PGPROC gains a single FutureWaitLock slot (locktag + mode). Only the
owning proc writes it; remote readers must hold the partition lock
covering the tag. The invariant "slot set implies not yet really
waiting for that (tag, mode)" is maintained by clearing the slot
inside LockAcquireExtended() at the point the proc attaches its
waitLink for that same request. InitProcess, ProcKill,
LockErrorCleanup, and LockReleaseAll also clear the slot.
* Fast-path holders of relation locks are invisible to the deadlock
detector, so CheckDeadLock() now snapshots the currently declared
fast-path-relevant future waits, bumps FastPathStrongRelationLocks
counters for those tags, and transfers existing fast-path holders
into the main lock table before walking the waits-for graph. A
shared atomic count lets the common case (no active declarations)
skip this work entirely. If a new fast-path-relevant declaration
appears between the snapshot and the partition-locked walk, the
check restarts with a fresh snapshot.
* FindLockCycleRecurseFuture() follows the declared edge to current
holders and to earlier queued waiters that JoinWaitQueue() would
place ahead of the future request. These edges are not recorded as
soft edges: the declarer is not in the wait queue, so reordering
cannot break such a cycle.
* DeadLockReport() emits "future deadlock detected" and a
"will request ... (declared future intent)" detail line when the
cycle involves a declared edge. SQLSTATE remains
ERRCODE_T_R_DEADLOCK_DETECTED.
New isolation permutations under src/test/modules/injection_points
cover the no-deadlock case (waiter with no conflicting held lock),
two two-backend future-deadlock cycles (held AccessShare + future
AccessExclusive / ShareUpdateExclusive), and a three-backend cycle
through an unrelated table.
---
src/backend/commands/repack.c | 4 +-
src/backend/commands/tablecmds.c | 33 ++
src/backend/storage/lmgr/deadlock.c | 195 ++++++++-
src/backend/storage/lmgr/lock.c | 410 ++++++++++++++++++
src/backend/storage/lmgr/proc.c | 36 ++
src/include/commands/tablecmds.h | 3 +
src/include/storage/lmgr.h | 2 +
src/include/storage/lock.h | 33 ++
src/include/storage/proc.h | 9 +
.../injection_points/expected/repack.out | 181 +++++++-
.../injection_points/specs/repack.spec | 118 +++++
11 files changed, 1018 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 58e3867246f..8261c58daca 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -2326,8 +2326,8 @@ process_single_relation(RepackStmt *stmt, LOCKMODE lockmode, bool isTopLevel,
tableOid = RangeVarGetRelidExtended(stmt->relation->relation,
lockmode,
0,
- RangeVarCallbackMaintainsTable,
- NULL);
+ RangeVarCallbackForRepack,
+ params);
rel = table_open(tableOid, NoLock);
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index eec09ba1ded..b08f57017e3 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -19761,6 +19761,39 @@ RangeVarCallbackMaintainsTable(const RangeVar *relation,
relation->relname);
}
+/*
+ * Callback to RangeVarGetRelidExtended() for REPACK. For concurrent repack,
+ * declare the future AccessExclusiveLock before the caller locks the table
+ * with ShareUpdateExclusiveLock, so deadlock checks can see the pending
+ * request while the weaker lock is held or awaited.
+ */
+void
+RangeVarCallbackForRepack(const RangeVar *relation,
+ Oid relId, Oid oldRelId, void *arg)
+{
+ ClusterParams *params = arg;
+ LOCKTAG locktag;
+ Oid dbid;
+
+ RangeVarCallbackMaintainsTable(relation, relId, oldRelId, NULL);
+
+ if ((params->options & CLUOPT_CONCURRENT) == 0)
+ return;
+
+ if (relId == oldRelId)
+ return;
+
+ if (OidIsValid(oldRelId))
+ LockClearFutureWaitSlot(false);
+
+ if (!OidIsValid(relId))
+ return;
+
+ dbid = IsSharedRelation(relId) ? InvalidOid : MyDatabaseId;
+ SET_LOCKTAG_RELATION(locktag, dbid, relId);
+ LockDeclareFutureWait(&locktag, AccessExclusiveLock);
+}
+
/*
* Callback to RangeVarGetRelidExtended() for TRUNCATE processing.
*/
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index b8962d875b6..a8ea9337e8d 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -74,6 +74,7 @@ typedef struct
LOCKTAG locktag; /* ID of awaited lock object */
LOCKMODE lockmode; /* type of lock we're waiting for */
int pid; /* PID of blocked backend */
+ bool is_future; /* was this a declared future-wait edge? */
} DEADLOCK_INFO;
@@ -86,6 +87,11 @@ static bool FindLockCycleRecurse(PGPROC *checkProc, int depth,
static bool FindLockCycleRecurseMember(PGPROC *checkProc,
PGPROC *checkProcLeader,
int depth, EDGE *softEdges, int *nSoftEdges);
+static bool FindLockCycleRecurseWait(PGPROC *checkProc,
+ PGPROC *checkProcLeader,
+ int depth, EDGE *softEdges, int *nSoftEdges);
+static bool FindLockCycleRecurseFuture(PGPROC *checkProc,
+ int depth, EDGE *softEdges, int *nSoftEdges);
static bool ExpandConstraints(EDGE *constraints, int nConstraints);
static bool TopoSort(LOCK *lock, EDGE *constraints, int nConstraints,
PGPROC **ordering);
@@ -504,7 +510,8 @@ FindLockCycleRecurse(PGPROC *checkProc,
* If the process is waiting, there is an outgoing waits-for edge to each
* process that blocks it.
*/
- if (!dlist_node_is_detached(&checkProc->waitLink) &&
+ if ((!dlist_node_is_detached(&checkProc->waitLink) ||
+ FutureWaitLockIsSet(&checkProc->futureWaitLock)) &&
FindLockCycleRecurseMember(checkProc, checkProc, depth, softEdges,
nSoftEdges))
return true;
@@ -538,6 +545,34 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
int depth,
EDGE *softEdges, /* output argument */
int *nSoftEdges) /* output argument */
+{
+ /* Follow outgoing edges from a real lock wait, if any. */
+ if (!dlist_node_is_detached(&checkProc->waitLink) &&
+ FindLockCycleRecurseWait(checkProc, checkProcLeader, depth,
+ softEdges, nSoftEdges))
+ return true;
+
+ /*
+ * A future-wait slot is only useful when another backend is already
+ * waiting and reaches this proc through the waits-for graph. Do not start
+ * from our own future slot: until we actually request that lock, there is
+ * no current wait to break.
+ */
+ if (checkProc != MyProc &&
+ checkProc == checkProcLeader &&
+ FutureWaitLockIsSet(&checkProc->futureWaitLock) &&
+ FindLockCycleRecurseFuture(checkProc, depth, softEdges, nSoftEdges))
+ return true;
+
+ return false;
+}
+
+static bool
+FindLockCycleRecurseWait(PGPROC *checkProc,
+ PGPROC *checkProcLeader,
+ int depth,
+ EDGE *softEdges, /* output argument */
+ int *nSoftEdges) /* output argument */
{
PGPROC *proc;
LOCK *lock = checkProc->waitLock;
@@ -590,6 +625,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ info->is_future = false;
return true;
}
@@ -679,6 +715,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ info->is_future = false;
/*
* Add this edge to the list of soft edges in the cycle
@@ -753,6 +790,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ info->is_future = false;
/*
* Add this edge to the list of soft edges in the cycle
@@ -774,6 +812,149 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
return false;
}
+/*
+ * Follow a declared future-wait edge as a hard edge.
+ */
+static bool
+FindLockCycleRecurseFuture(PGPROC *checkProc,
+ int depth,
+ EDGE *softEdges, /* output argument */
+ int *nSoftEdges) /* output argument */
+{
+ FutureWaitLock *futureWaitLock = &checkProc->futureWaitLock;
+ LOCK *lock;
+ LockMethod lockMethodTable;
+ uint32 hashcode;
+ int conflictMask;
+ LOCKMASK myHeldLocks = 0;
+ dlist_iter proclock_iter;
+ dlist_iter proc_iter;
+
+ Assert(FutureWaitLockIsSet(futureWaitLock));
+ Assert(checkProc != MyProc);
+
+ /*
+ * Invariant maintained by LockAcquireExtended(): if a proc is really
+ * waiting on a lock, its future-wait slot (if any) must not describe the
+ * same (locktag, lockmode). The slot is cleared at the point we attach
+ * our waitLink for that exact lock and mode, so the wait walker's
+ * procLocks scan already subsumes anything this walker would find.
+ */
+ Assert(dlist_node_is_detached(&checkProc->waitLink) ||
+ checkProc->waitLockMode != futureWaitLock->mode ||
+ memcmp(&checkProc->waitLock->tag, &futureWaitLock->locktag,
+ sizeof(LOCKTAG)) != 0);
+
+ /*
+ * Look up the shared LOCK object for the declared tag. It is legal for
+ * the entry not to exist: the hash table only contains LOCK objects that
+ * currently have at least one holder or requester, and a future-wait
+ * declaration can be made before any proclock for that tag is created
+ * (e.g. RangeVarCallbackForRepack runs inside RangeVarGetRelidExtended
+ * *before* the caller has actually taken the initial SUE lock on the
+ * relation, and no other backend may have any lock on it either). With
+ * no LOCK object there are no current holders, hence no outgoing edges
+ * from this future-wait slot, so simply return.
+ */
+ hashcode = LockTagHashCode(&futureWaitLock->locktag);
+ lock = LockHashLookup(&futureWaitLock->locktag, hashcode);
+ if (lock == NULL)
+ return false;
+
+ lockMethodTable = GetLocksMethodTable(lock);
+ Assert(futureWaitLock->mode > 0 &&
+ futureWaitLock->mode <= lockMethodTable->numLockModes);
+ conflictMask = lockMethodTable->conflictTab[futureWaitLock->mode];
+
+ /*
+ * First, follow edges to current holders of conflicting modes. These
+ * edges are the future-wait equivalent of the normal hard edges from a
+ * real waiter to current holders.
+ */
+ dlist_foreach(proclock_iter, &lock->procLocks)
+ {
+ PROCLOCK *proclock = dlist_container(PROCLOCK, lockLink, proclock_iter.cur);
+ PGPROC *proc = proclock->tag.myProc;
+ PGPROC *leader;
+
+ leader = proc->lockGroupLeader == NULL ? proc : proc->lockGroupLeader;
+
+ /* A proc never blocks itself or any other lock group member. */
+ if (leader == checkProc)
+ {
+ myHeldLocks |= proclock->holdMask;
+ continue;
+ }
+
+ if ((proclock->holdMask & conflictMask) != 0)
+ {
+ if (FindLockCycleRecurse(proc, depth + 1,
+ softEdges, nSoftEdges))
+ {
+ DEADLOCK_INFO *info = &deadlockDetails[depth];
+
+ info->locktag = futureWaitLock->locktag;
+ info->lockmode = futureWaitLock->mode;
+ info->pid = checkProc->pid;
+ info->is_future = true;
+
+ return true;
+ }
+ }
+ }
+
+ /*
+ * Existing waiters can also become blockers for the future request. A
+ * real LockAcquireExtended() first checks lock->waitMask, and if there
+ * are conflicting waiters it calls JoinWaitQueue(). Model the queue
+ * position JoinWaitQueue() would choose today, and follow only queued
+ * requests that would be ahead of that position.
+ *
+ * Do not record these as soft edges. The future requester is not present
+ * in lock->waitProcs, so the deadlock detector cannot repair such an edge
+ * by reordering the current wait queue.
+ */
+ dclist_foreach(proc_iter, &lock->waitProcs)
+ {
+ PGPROC *proc = dlist_container(PGPROC, waitLink, proc_iter.cur);
+ PGPROC *leader;
+ LOCKMODE waitLockMode = proc->waitLockMode;
+
+ leader = proc->lockGroupLeader == NULL ? proc : proc->lockGroupLeader;
+
+ if (leader == checkProc)
+ continue;
+
+ /*
+ * If checkProc already holds locks that conflict with this waiter's
+ * request, JoinWaitQueue() would insert the future request before
+ * this waiter. We are done scanning the queue after considering
+ * earlier waiters.
+ */
+ if (myHeldLocks != 0 &&
+ (lockMethodTable->conflictTab[waitLockMode] & myHeldLocks) != 0)
+ break;
+
+ if ((LOCKBIT_ON(waitLockMode) & conflictMask) != 0)
+ {
+ if (FindLockCycleRecurse(proc, depth + 1,
+ softEdges, nSoftEdges))
+ {
+ DEADLOCK_INFO *info = &deadlockDetails[depth];
+
+ info->locktag = futureWaitLock->locktag;
+ info->lockmode = futureWaitLock->mode;
+ info->pid = checkProc->pid;
+ info->is_future = true;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
/*
* ExpandConstraints -- expand a list of constraints into a set of
@@ -1078,6 +1259,7 @@ DeadLockReport(void)
StringInfoData logbuf; /* errdetail for server log */
StringInfoData locktagbuf;
int i;
+ bool any_future = false;
initStringInfo(&clientbuf);
initStringInfo(&logbuf);
@@ -1099,12 +1281,15 @@ DeadLockReport(void)
resetStringInfo(&locktagbuf);
DescribeLockTag(&locktagbuf, &info->locktag);
+ any_future |= info->is_future;
if (i > 0)
appendStringInfoChar(&clientbuf, '\n');
appendStringInfo(&clientbuf,
- _("Process %d waits for %s on %s; blocked by process %d."),
+ info->is_future
+ ? _("Process %d will request %s on %s (declared future intent); blocked by process %d.")
+ : _("Process %d waits for %s on %s; blocked by process %d."),
info->pid,
GetLockmodeName(info->locktag.locktag_lockmethodid,
info->lockmode),
@@ -1132,7 +1317,9 @@ DeadLockReport(void)
ereport(ERROR,
(errcode(ERRCODE_T_R_DEADLOCK_DETECTED),
- errmsg("deadlock detected"),
+ errmsg(any_future
+ ? "future deadlock detected"
+ : "deadlock detected"),
errdetail_internal("%s", clientbuf.data),
errdetail_log("%s", logbuf.data),
errhint("See server log for query details.")));
@@ -1154,9 +1341,11 @@ RememberSimpleDeadLock(PGPROC *proc1,
info->locktag = lock->tag;
info->lockmode = lockmode;
info->pid = proc1->pid;
+ info->is_future = false;
info++;
info->locktag = proc2->waitLock->tag;
info->lockmode = proc2->waitLockMode;
info->pid = proc2->pid;
+ info->is_future = false;
nDeadlockDetails = 2;
}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index c221fe96889..12107e07bec 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -314,6 +314,19 @@ typedef struct
static volatile FastPathStrongRelationLockData *FastPathStrongRelationLocks;
+/*
+ * Count of active future-wait declarations that could conflict with relation
+ * fast-path locks. This is only a skip hint for CheckDeadLock(); the actual
+ * future-wait locktags are read from PGPROC.futureWaitLock while all lock
+ * partitions are held.
+ */
+typedef struct FutureWaitFastPathCtlData
+{
+ pg_atomic_uint32 count;
+} FutureWaitFastPathCtlData;
+
+static FutureWaitFastPathCtlData *FutureWaitFastPathCtl = NULL;
+
static void LockManagerShmemRequest(void *arg);
static void LockManagerShmemInit(void *arg);
@@ -488,12 +501,18 @@ LockManagerShmemRequest(void *arg)
.size = sizeof(FastPathStrongRelationLockData),
.ptr = (void **) (void *) &FastPathStrongRelationLocks,
);
+
+ ShmemRequestStruct(.name = "Future Wait Fast Path Control",
+ .size = sizeof(FutureWaitFastPathCtlData),
+ .ptr = (void **) &FutureWaitFastPathCtl,
+ );
}
static void
LockManagerShmemInit(void *arg)
{
SpinLockInit(&FastPathStrongRelationLocks->mutex);
+ pg_atomic_init_u32(&FutureWaitFastPathCtl->count, 0);
}
/*
@@ -628,6 +647,374 @@ DoLockModesConflict(LOCKMODE mode1, LOCKMODE mode2)
return false;
}
+/*
+ * LockHashLookup -- look up a LOCK by locktag and precomputed hashcode.
+ *
+ * Returns NULL if no LOCK exists for this tag.
+ *
+ * Callers MUST hold the partition lock covering hashcode in LW_EXCLUSIVE
+ * mode, and MUST treat the returned pointer as valid only while that
+ * partition lock remains held: the LOCK entry can be removed from the hash
+ * table by LockRelease() under the same partition lock, so dropping the
+ * lock invalidates the pointer.
+ *
+ * Today the only caller is the deadlock detector, which satisfies both
+ * requirements by acquiring every lock-manager partition lock exclusively
+ * before walking the wait-for graph, and uses the result to resolve a
+ * declared future-wait edge's locktag into a live LOCK *. New callers
+ * from outside that context should be reviewed carefully.
+ */
+LOCK *
+LockHashLookup(const LOCKTAG *locktag, uint32 hashcode)
+{
+ Assert(LWLockHeldByMeInMode(LockHashPartitionLock(hashcode),
+ LW_EXCLUSIVE));
+
+ return (LOCK *) hash_search_with_hash_value(LockMethodLockHash,
+ locktag,
+ hashcode,
+ HASH_FIND,
+ NULL);
+}
+
+static inline bool
+FutureWaitLockMatches(const FutureWaitLock *futureWaitLock,
+ const LOCKTAG *locktag, LOCKMODE lockmode)
+{
+ return FutureWaitLockIsSet(futureWaitLock) &&
+ futureWaitLock->mode == lockmode &&
+ memcmp(&futureWaitLock->locktag, locktag, sizeof(LOCKTAG)) == 0;
+}
+
+static inline bool
+FutureWaitNeedsFastPathMigration(const FutureWaitLock *futureWaitLock)
+{
+ return FutureWaitLockIsSet(futureWaitLock) &&
+ ConflictsWithRelationFastPath(&futureWaitLock->locktag,
+ futureWaitLock->mode);
+}
+
+/*
+ * LockClearFutureWaitSlotIfMatch -- clear our future-wait slot if it matches
+ * a lock that we have now acquired or started waiting for.
+ */
+static void
+LockClearFutureWaitSlotIfMatch(const LOCKTAG *locktag, LOCKMODE lockmode,
+ bool partitionLockHeld)
+{
+ if (!FutureWaitLockMatches(&MyProc->futureWaitLock, locktag, lockmode))
+ return;
+
+ LockClearFutureWaitSlot(partitionLockHeld);
+}
+
+/*
+ * LockDeclareFutureWait -- publish a hard future waits-for edge.
+ *
+ * The caller declares that this backend intends to request lockmode on
+ * locktag later. The declaration becomes visible to deadlock detection
+ * before the real lock acquisition happens.
+ *
+ * Only one future-wait slot is supported per backend, and parallel workers
+ * (non-leader lock-group members) cannot declare a future wait. REPACK
+ * CONCURRENTLY is the only current caller and satisfies both constraints.
+ */
+void
+LockDeclareFutureWait(const LOCKTAG *locktag, LOCKMODE lockmode)
+{
+ uint32 hashcode;
+ LWLock *partitionLock;
+
+ Assert(MyProc != NULL);
+ Assert(MyProc->lockGroupLeader == NULL ||
+ MyProc->lockGroupLeader == MyProc);
+ Assert(!LockHeldByMe(locktag, lockmode, true));
+ Assert(FutureWaitLockIsEmpty(&MyProc->futureWaitLock));
+
+ hashcode = LockTagHashCode(locktag);
+ partitionLock = LockHashPartitionLock(hashcode);
+
+ /*
+ * Bump the fast-path migration hint before publishing the slot. A
+ * concurrent CheckDeadLock() may see the hint before the slot is visible,
+ * which is harmless; the reverse ordering would allow it to observe a
+ * future edge without migrating fast-path holders for that relation.
+ */
+ if (ConflictsWithRelationFastPath(locktag, lockmode))
+ pg_atomic_fetch_add_u32(&FutureWaitFastPathCtl->count, 1);
+
+ LWLockAcquire(partitionLock, LW_EXCLUSIVE);
+
+ MyProc->futureWaitLock.locktag = *locktag;
+ MyProc->futureWaitLock.mode = lockmode;
+
+ LWLockRelease(partitionLock);
+}
+
+/*
+ * LockClearFutureWaitSlot -- clear this backend's declared future wait, if any.
+ */
+void
+LockClearFutureWaitSlot(bool partitionLockHeld)
+{
+ LWLock *partitionLock;
+ LOCKTAG clearedTag;
+ LOCKMODE clearedMode;
+ bool cleared = false;
+
+ if (MyProc == NULL || FutureWaitLockIsEmpty(&MyProc->futureWaitLock))
+ return;
+
+ partitionLock =
+ LockHashPartitionLock(LockTagHashCode(&MyProc->futureWaitLock.locktag));
+ if (!partitionLockHeld)
+ LWLockAcquire(partitionLock, LW_EXCLUSIVE);
+ else
+ Assert(LWLockHeldByMeInMode(partitionLock, LW_EXCLUSIVE));
+
+ if (FutureWaitLockIsSet(&MyProc->futureWaitLock))
+ {
+ clearedTag = MyProc->futureWaitLock.locktag;
+ clearedMode = MyProc->futureWaitLock.mode;
+ MemSet(&MyProc->futureWaitLock, 0, sizeof(FutureWaitLock));
+ cleared = true;
+ }
+
+ if (!partitionLockHeld)
+ LWLockRelease(partitionLock);
+
+ if (cleared && ConflictsWithRelationFastPath(&clearedTag, clearedMode))
+ {
+ Assert(pg_atomic_read_u32(&FutureWaitFastPathCtl->count) > 0);
+ pg_atomic_fetch_sub_u32(&FutureWaitFastPathCtl->count, 1);
+ }
+}
+
+/*
+ * FutureWaitLocktagSnapshotContains -- linear membership test over a
+ * locktag snapshot.
+ *
+ * n is bounded by the number of declared future waits (effectively the
+ * number of concurrent REPACK CONCURRENTLY commands), which in practice
+ * is 0 or 1, so a linear scan is cheaper than building a hash or keeping
+ * the array sorted.
+ */
+static bool
+FutureWaitLocktagSnapshotContains(LOCKTAG *locktags, int n,
+ const LOCKTAG *locktag)
+{
+ for (int i = 0; i < n; i++)
+ {
+ if (memcmp(&locktags[i], locktag, sizeof(LOCKTAG)) == 0)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * SnapshotFutureWaitFastPathLocks -- copy the current fast-path-relevant
+ * future-wait tags while all lock partitions are held.
+ *
+ * The caller holds every lock-manager partition lock exclusively, so this
+ * function must not take any LWLock. palloc is safe here because the
+ * memory context machinery does not take LWLocks; a palloc failure would
+ * ereport out of the deadlock check but not deadlock against itself.
+ */
+static LOCKTAG *
+SnapshotFutureWaitFastPathLocks(int *out_count)
+{
+ LOCKTAG *locktags;
+ int n = 0;
+
+ *out_count = 0;
+
+ locktags = palloc_array(LOCKTAG, ProcGlobal->allProcCount);
+
+ for (int i = 0; i < ProcGlobal->allProcCount; i++)
+ {
+ PGPROC *proc = GetPGProcByNumber(i);
+ FutureWaitLock *futureWaitLock = &proc->futureWaitLock;
+
+ if (!FutureWaitNeedsFastPathMigration(futureWaitLock))
+ continue;
+
+ Assert(LWLockHeldByMeInMode(LockHashPartitionLock(
+ LockTagHashCode(&futureWaitLock->locktag)),
+ LW_EXCLUSIVE));
+
+ if (!FutureWaitLocktagSnapshotContains(locktags, n,
+ &futureWaitLock->locktag))
+ {
+ Assert(n < ProcGlobal->allProcCount);
+ locktags[n++] = futureWaitLock->locktag;
+ }
+ }
+
+ if (n == 0)
+ {
+ pfree(locktags);
+ return NULL;
+ }
+
+ *out_count = n;
+ return locktags;
+}
+
+/*
+ * MigrateFutureWaitFastPathLocks -- make fast-path holders visible to the
+ * deadlock detector for currently declared future waits.
+ *
+ * Future-wait declarations do not take a real strong lock yet, so they do not
+ * trigger the normal fast-path transfer performed by LockAcquireExtended().
+ * This helper takes a snapshot of current fast-path-relevant future waits,
+ * installs temporary strong-lock counter bumps for those relation hash
+ * partitions, and transfers existing fast-path holders of those relations
+ * into the main lock table.
+ *
+ * The returned snapshot must remain active until CheckDeadLock() has finished
+ * its partition-locked graph walk. New fast-path holders cannot appear for
+ * migrated tags while the temporary strong-lock counter bumps are active. If
+ * a new future-wait tag appears after this snapshot, CheckDeadLock() detects
+ * that with FutureWaitFastPathSnapshotCoversCurrentLocks() and retries.
+ */
+LOCKTAG *
+MigrateFutureWaitFastPathLocks(int *out_count)
+{
+ LockMethod lockMethodTable = LockMethods[DEFAULT_LOCKMETHOD];
+ LOCKTAG *locktags;
+ int n;
+
+ *out_count = 0;
+
+ /* Cheap path: no active declaration can involve fast-path holders. */
+ if (pg_atomic_read_u32(&FutureWaitFastPathCtl->count) == 0)
+ return NULL;
+
+ for (int i = 0; i < NUM_LOCK_PARTITIONS; i++)
+ LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);
+
+ locktags = SnapshotFutureWaitFastPathLocks(&n);
+
+ for (int i = NUM_LOCK_PARTITIONS; --i >= 0;)
+ LWLockRelease(LockHashPartitionLockByIndex(i));
+
+ if (locktags == NULL)
+ return NULL;
+
+ /*
+ * Block new fast-path acquisitions for this snapshot before transferring
+ * existing holders to the main table.
+ */
+ SpinLockAcquire(&FastPathStrongRelationLocks->mutex);
+ for (int i = 0; i < n; i++)
+ {
+ uint32 hashcode = LockTagHashCode(&locktags[i]);
+ uint32 fasthashcode = FastPathStrongLockHashPartition(hashcode);
+
+ FastPathStrongRelationLocks->count[fasthashcode]++;
+ }
+ SpinLockRelease(&FastPathStrongRelationLocks->mutex);
+
+ for (int i = 0; i < n; i++)
+ {
+ uint32 hashcode = LockTagHashCode(&locktags[i]);
+
+ if (!FastPathTransferRelationLocks(lockMethodTable, &locktags[i],
+ hashcode))
+ {
+ SpinLockAcquire(&FastPathStrongRelationLocks->mutex);
+ for (int j = 0; j < n; j++)
+ {
+ uint32 restorehashcode = LockTagHashCode(&locktags[j]);
+ uint32 restorefasthashcode =
+ FastPathStrongLockHashPartition(restorehashcode);
+
+ Assert(FastPathStrongRelationLocks->count[restorefasthashcode]
+ > 0);
+ FastPathStrongRelationLocks->count[restorefasthashcode]--;
+ }
+ SpinLockRelease(&FastPathStrongRelationLocks->mutex);
+ pfree(locktags);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of shared memory"),
+ errhint("You might need to increase \"%s\".", "max_locks_per_transaction")));
+ }
+ }
+
+ *out_count = n;
+ return locktags;
+}
+
+/*
+ * FutureWaitFastPathSnapshotCoversCurrentLocks -- does the active snapshot
+ * cover all fast-path-relevant future waits currently visible to the deadlock
+ * detector?
+ *
+ * Caller must hold all lock partitions. If this returns false, a declaration
+ * raced with the previous snapshot and CheckDeadLock() must retry migration
+ * before walking the waits-for graph.
+ */
+bool
+FutureWaitFastPathSnapshotCoversCurrentLocks(LOCKTAG *locktags, int n)
+{
+ /*
+ * With no snapshot and no in-progress fast-path-relevant declaration,
+ * there cannot be anything to validate. If the counter is nonzero, still
+ * scan: a declaration may have incremented the counter before publishing
+ * its slot, causing the snapshot to be empty.
+ */
+ if (locktags == NULL &&
+ pg_atomic_read_u32(&FutureWaitFastPathCtl->count) == 0)
+ return true;
+
+ for (int i = 0; i < ProcGlobal->allProcCount; i++)
+ {
+ PGPROC *proc = GetPGProcByNumber(i);
+ FutureWaitLock *futureWaitLock = &proc->futureWaitLock;
+
+ if (!FutureWaitNeedsFastPathMigration(futureWaitLock))
+ continue;
+
+ Assert(LWLockHeldByMeInMode(LockHashPartitionLock(
+ LockTagHashCode(&futureWaitLock->locktag)),
+ LW_EXCLUSIVE));
+
+ if (!FutureWaitLocktagSnapshotContains(locktags, n,
+ &futureWaitLock->locktag))
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * RestoreFutureWaitFastPathSnapshot -- undo the fast-path strong-lock counter
+ * bumps for a snapshot returned by MigrateFutureWaitFastPathLocks().
+ *
+ * Must be called after CheckDeadLock() has released all partition locks.
+ */
+void
+RestoreFutureWaitFastPathSnapshot(LOCKTAG *locktags, int n)
+{
+ if (locktags == NULL)
+ return;
+
+ SpinLockAcquire(&FastPathStrongRelationLocks->mutex);
+ for (int i = 0; i < n; i++)
+ {
+ uint32 hashcode = LockTagHashCode(&locktags[i]);
+ uint32 fasthashcode = FastPathStrongLockHashPartition(hashcode);
+
+ Assert(FastPathStrongRelationLocks->count[fasthashcode] > 0);
+ FastPathStrongRelationLocks->count[fasthashcode]--;
+ }
+ SpinLockRelease(&FastPathStrongRelationLocks->mutex);
+
+ pfree(locktags);
+}
+
/*
* LockHeldByMe -- test whether lock 'locktag' is held by the current
* transaction
@@ -937,6 +1324,8 @@ LockAcquireExtended(const LOCKTAG *locktag,
*/
if (locallock->nLocks > 0)
{
+ Assert(!FutureWaitLockMatches(&MyProc->futureWaitLock, locktag,
+ lockmode));
GrantLockLocal(locallock, owner);
if (locallock->lockCleared)
return LOCKACQUIRE_ALREADY_CLEAR;
@@ -1012,6 +1401,7 @@ LockAcquireExtended(const LOCKTAG *locktag,
*/
locallock->lock = NULL;
locallock->proclock = NULL;
+ LockClearFutureWaitSlotIfMatch(locktag, lockmode, false);
GrantLockLocal(locallock, owner);
return LOCKACQUIRE_OK;
}
@@ -1225,6 +1615,20 @@ LockAcquireExtended(const LOCKTAG *locktag,
Assert(!dontWait);
PROCLOCK_PRINT("LockAcquire: sleeping on lock", proclock);
LOCK_PRINT("LockAcquire: sleeping on lock", lock, lockmode);
+
+ /*
+ * We have attached waitLink and are about to sleep on exactly the
+ * (locktag, lockmode) we had declared as a future wait. Remote
+ * walkers will now see us as a real waiter on this lock, and
+ * FindLockCycleRecurseWait() will scan lock->procLocks with the same
+ * conflictMask the future walker would have used. Clearing the
+ * future slot now, while we still hold the partition lock, keeps
+ * the invariant "future slot set means not yet really waiting for the
+ * same lock and mode" and removes duplicate work from the deadlock
+ * detector.
+ */
+ LockClearFutureWaitSlotIfMatch(locktag, lockmode, true);
+
LWLockRelease(partitionLock);
waitResult = WaitOnLock(locallock, owner);
@@ -1247,7 +1651,10 @@ LockAcquireExtended(const LOCKTAG *locktag,
}
}
else
+ {
+ LockClearFutureWaitSlotIfMatch(locktag, lockmode, true);
LWLockRelease(partitionLock);
+ }
Assert(waitResult == PROC_WAIT_STATUS_OK);
/* The lock was granted to us. Update the local lock entry accordingly */
@@ -2328,6 +2735,9 @@ LockReleaseAll(LOCKMETHODID lockmethodid, bool allLocks)
elog(ERROR, "unrecognized lock method: %d", lockmethodid);
lockMethodTable = LockMethods[lockmethodid];
+ if (lockmethodid == DEFAULT_LOCKMETHOD)
+ LockClearFutureWaitSlot(false);
+
#ifdef LOCK_DEBUG
if (*(lockMethodTable->trace_flag))
elog(LOG, "LockReleaseAll: lockmethod=%d", lockmethodid);
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1ac25068d62..b6b9de2d51c 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -494,6 +494,7 @@ InitProcess(void)
MyProc->waitLock = NULL;
dlist_node_init(&MyProc->waitLink);
MyProc->waitProcLock = NULL;
+ MemSet(&MyProc->futureWaitLock, 0, sizeof(FutureWaitLock));
pg_atomic_write_u64(&MyProc->waitStart, 0);
#ifdef USE_ASSERT_CHECKING
{
@@ -691,6 +692,7 @@ InitAuxiliaryProcess(void)
MyProc->waitLock = NULL;
dlist_node_init(&MyProc->waitLink);
MyProc->waitProcLock = NULL;
+ MemSet(&MyProc->futureWaitLock, 0, sizeof(FutureWaitLock));
pg_atomic_write_u64(&MyProc->waitStart, 0);
#ifdef USE_ASSERT_CHECKING
{
@@ -824,6 +826,7 @@ LockErrorCleanup(void)
HOLD_INTERRUPTS();
AbortStrongLockAcquire();
+ LockClearFutureWaitSlot(false);
/* Nothing to do if we weren't waiting for a lock */
lockAwaited = GetAwaitedLock();
@@ -934,6 +937,7 @@ ProcKill(int code, Datum arg)
/* Make sure we're out of the sync rep lists */
SyncRepCleanupAtProcExit();
+ LockClearFutureWaitSlot(false);
#ifdef USE_ASSERT_CHECKING
{
@@ -1824,6 +1828,17 @@ CheckDeadLock(void)
{
int i;
DeadLockState result;
+ LOCKTAG *futureLocks;
+ int futureLockCount = 0;
+
+retry:
+ /*
+ * Migrate fast-path holders for relation locks named by declared
+ * future-wait declarations. The helper takes and releases lock
+ * partitions internally while taking its snapshot, so it must run before
+ * this routine freezes the lock table for the real deadlock check.
+ */
+ futureLocks = MigrateFutureWaitFastPathLocks(&futureLockCount);
/*
* Acquire exclusive lock on the entire shared lock data structures. Must
@@ -1838,6 +1853,25 @@ CheckDeadLock(void)
for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);
+ /*
+ * A future-wait declaration can appear after the migration snapshot but
+ * before this partition-locked graph walk. If so, release the temporary
+ * strong-lock counts and retry, so the newly visible future edge cannot
+ * miss holders that are still in fast-path arrays.
+ *
+ * In theory a stream of concurrent declarations could force repeated
+ * retries, but in practice future-wait declarations are issued only by
+ * REPACK CONCURRENTLY and are rare, so the loop terminates quickly.
+ */
+ if (!FutureWaitFastPathSnapshotCoversCurrentLocks(futureLocks,
+ futureLockCount))
+ {
+ for (i = NUM_LOCK_PARTITIONS; --i >= 0;)
+ LWLockRelease(LockHashPartitionLockByIndex(i));
+ RestoreFutureWaitFastPathSnapshot(futureLocks, futureLockCount);
+ goto retry;
+ }
+
/*
* Check to see if we've been awoken by anyone in the interim.
*
@@ -1902,6 +1936,8 @@ check_done:
for (i = NUM_LOCK_PARTITIONS; --i >= 0;)
LWLockRelease(LockHashPartitionLockByIndex(i));
+ RestoreFutureWaitFastPathSnapshot(futureLocks, futureLockCount);
+
return result;
}
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index c3d8518cb62..648435eb618 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -102,6 +102,9 @@ extern void AtEOSubXact_on_commit_actions(bool isCommit,
extern void RangeVarCallbackMaintainsTable(const RangeVar *relation,
Oid relId, Oid oldRelId,
void *arg);
+extern void RangeVarCallbackForRepack(const RangeVar *relation,
+ Oid relId, Oid oldRelId,
+ void *arg);
extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 2a985ce5e15..b301edafb31 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -51,6 +51,8 @@ extern bool CheckRelationLockedByMe(Relation relation, LOCKMODE lockmode,
extern bool CheckRelationOidLockedByMe(Oid relid, LOCKMODE lockmode,
bool orstronger);
extern bool LockHasWaitersRelation(Relation relation, LOCKMODE lockmode);
+extern void LockDeclareFutureWait(const LOCKTAG *locktag, LOCKMODE lockmode);
+extern void LockClearFutureWaitSlot(bool partitionLockHeld);
extern void LockRelationIdForSession(LockRelId *relid, LOCKMODE lockmode);
extern void UnlockRelationIdForSession(LockRelId *relid, LOCKMODE lockmode);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index ee3cb1dc203..dca5a3d33d6 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -275,6 +275,30 @@ typedef struct LOCALLOCK
#define LOCALLOCK_LOCKTAG(llock) ((LockTagType) (llock).tag.lock.locktag_type)
+/*
+ * Declared "future wait" for a lock the backend intends to acquire later.
+ *
+ * A process may publish its intent to acquire a particular lock mode on a
+ * particular locktag before actually calling LockAcquire(). The deadlock
+ * detector treats a populated slot as a hard waits-for edge from the
+ * declarer to every current holder whose mode conflicts with .mode. The
+ * slot is empty when locktag.locktag_lockmethodid == 0.
+ *
+ * Grouped in a struct so a future change can turn the slot into a short
+ * list without touching every reader.
+ */
+typedef struct FutureWaitLock
+{
+ LOCKTAG locktag; /* locktag_lockmethodid == 0 when empty */
+ LOCKMODE mode; /* mode the proc will eventually request */
+} FutureWaitLock;
+
+#define FutureWaitLockIsSet(futureWaitLock) \
+ ((futureWaitLock)->locktag.locktag_lockmethodid != 0)
+#define FutureWaitLockIsEmpty(futureWaitLock) \
+ (!FutureWaitLockIsSet(futureWaitLock))
+
+
/*
* These structures hold information passed from lmgr internals to the lock
* listing user-level functions (in lockfuncs.c).
@@ -419,6 +443,15 @@ extern LOCALLOCK *GetAwaitedLock(void);
extern void ResetAwaitedLock(void);
extern void RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode);
+
+extern LOCK *LockHashLookup(const LOCKTAG *locktag, uint32 hashcode);
+
+/* Future-wait fast-path migration, called by CheckDeadLock(). */
+extern LOCKTAG *MigrateFutureWaitFastPathLocks(int *out_count);
+extern bool FutureWaitFastPathSnapshotCoversCurrentLocks(LOCKTAG *locktags,
+ int n);
+extern void RestoreFutureWaitFastPathSnapshot(LOCKTAG *locktags, int n);
+
extern LockData *GetLockStatusData(void);
extern BlockedProcsData *GetBlockerStatusData(int blocked_pid);
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 3e1d1fad5f9..66bfce6f8b4 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -314,6 +314,15 @@ typedef struct PGPROC
LOCKMASK heldLocks; /* bitmask for lock types already held on this
* lock object by this backend */
+ /*
+ * Declared future-wait slot: a lock this proc intends to acquire later.
+ * Empty when .locktag.locktag_lockmethodid == 0. Publishing, clearing,
+ * and remote reads are protected by the partition lock of .locktag
+ * (deadlock walkers hold all partition locks and can safely inspect the
+ * slot). The owning backend may inspect its own slot locklessly.
+ */
+ FutureWaitLock futureWaitLock;
+
pg_atomic_uint64 waitStart; /* time at which wait for lock acquisition
* started */
diff --git a/src/test/modules/injection_points/expected/repack.out b/src/test/modules/injection_points/expected/repack.out
index b575e9052ee..22c051f702e 100644
--- a/src/test/modules/injection_points/expected/repack.out
+++ b/src/test/modules/injection_points/expected/repack.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 4 sessions
starting permutation: wait_before_lock change_existing change_new change_subxact1 change_subxact2 check2 wakeup_before_lock check1
injection_points_attach
@@ -111,3 +111,182 @@ injection_points_detach
(1 row)
+
+starting permutation: wait_before_lock begin_txn lock_table_ae s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_txn:
+ BEGIN;
+
+step lock_table_ae:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step lock_table_ae: <... completed>
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read lock_table_ae end_txn s3_wakeup
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table_ae:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step lock_table_ae: <... completed>
+ERROR: future deadlock detected
+step end_txn:
+ COMMIT;
+
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read lock_table_sue end_txn s3_wakeup
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table_sue:
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step lock_table_sue: <... completed>
+ERROR: future deadlock detected
+step end_txn:
+ COMMIT;
+
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read s3_begin_txn s3_lock_table_y_sue s3_lock_table_sue lock_table_y_sue end_txn s4_wakeup s3_end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_begin_txn:
+ BEGIN;
+
+step s3_lock_table_y_sue:
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+
+step s3_lock_table_sue:
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step lock_table_y_sue:
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step lock_table_y_sue: <... completed>
+ERROR: future deadlock detected
+step end_txn:
+ COMMIT;
+
+step s4_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step s3_lock_table_sue: <... completed>
+step s3_end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/specs/repack.spec b/src/test/modules/injection_points/specs/repack.spec
index d727a9b056b..d214821a2da 100644
--- a/src/test/modules/injection_points/specs/repack.spec
+++ b/src/test/modules/injection_points/specs/repack.spec
@@ -5,6 +5,7 @@ setup
CREATE TABLE repack_test(i int PRIMARY KEY, j int);
INSERT INTO repack_test(i, j) VALUES (1, 1), (2, 2), (3, 3), (4, 4);
+ CREATE TABLE repack_test_y(i int);
CREATE TABLE relfilenodes(node oid);
@@ -15,6 +16,7 @@ setup
teardown
{
DROP TABLE repack_test;
+ DROP TABLE repack_test_y;
DROP EXTENSION injection_points;
DROP TABLE relfilenodes;
@@ -61,6 +63,10 @@ teardown
}
session s2
+step s2_timeout_fast
+{
+ SET deadlock_timeout = '10ms';
+}
# Change the existing data. UPDATE changes both key and non-key columns. Also
# update one row twice to test whether tuple version generated by this session
# can be found.
@@ -128,6 +134,60 @@ step wakeup_before_lock
{
SELECT injection_points_wakeup('repack-concurrently-before-lock');
}
+# Steps used in lock contention tests.
+step begin_txn
+{
+ BEGIN;
+}
+step begin_and_read
+{
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+}
+step lock_table_ae
+{
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+}
+step lock_table_sue
+{
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step lock_table_y_sue
+{
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step end_txn
+{
+ COMMIT;
+}
+
+session s3
+step s3_begin_txn
+{
+ BEGIN;
+}
+step s3_lock_table_y_sue
+{
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step s3_lock_table_sue
+{
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step s3_wakeup
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
+step s3_end_txn
+{
+ COMMIT;
+}
+
+session s4
+step s4_wakeup
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
# Test if data changes introduced while one session is performing REPACK
# CONCURRENTLY find their way into the table.
@@ -140,3 +200,61 @@ permutation
check2
wakeup_before_lock
check1
+
+# A waiter that does not already hold a conflicting lock on the table is not a
+# future deadlock. It waits until REPACK finishes and then acquires its lock.
+permutation
+ wait_before_lock
+ begin_txn
+ lock_table_ae
+ s3_wakeup
+ end_txn
+
+# In the deadlock-expecting permutations below, all deadlock detections must
+# run while s1 is still parked at the injection point, i.e. before s1 wakes
+# up and attempts AccessExclusiveLock. Once s1 attaches waitLink for AEL,
+# it clears its future-wait slot, and any cycle from that point on would be
+# reported as a plain "deadlock detected" rather than "future deadlock
+# detected". To force this ordering, every wakeup step is placed after the
+# COMMIT from the session whose 10 ms deadlock_timeout we rely on: the
+# framework cannot run that COMMIT until the blocked session has unblocked
+# (via its deadlock check), so the wakeup necessarily follows.
+
+# A waiter that already holds AccessShareLock then waits for AccessExclusiveLock
+# behind REPACK's ShareUpdateExclusiveLock.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ lock_table_ae(*)
+ end_txn
+ s3_wakeup
+
+# Same shape as above, but the waiter requests ShareUpdateExclusiveLock.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ lock_table_sue(*)
+ end_txn
+ s3_wakeup
+
+# Three-backend future deadlock:
+#
+# - s1 holds ShareUpdateExclusiveLock on repack_test and has declared a future
+# AccessExclusiveLock on it.
+# - s2 holds AccessShareLock on repack_test.
+# - s3 holds ShareUpdateExclusiveLock on repack_test_y, then waits for
+# ShareUpdateExclusiveLock on repack_test behind s1.
+# - s2 waits for ShareUpdateExclusiveLock on repack_test_y behind s3.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ s3_begin_txn
+ s3_lock_table_y_sue
+ s3_lock_table_sue
+ lock_table_y_sue(*)
+ end_txn
+ s4_wakeup
+ s3_end_txn
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-18 19:23 ` Antonin Houska <[email protected]>
2026-04-18 22:46 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-19 11:00 ` Re: Adding REPACK [concurrently] Alexander Lakhin <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-18 19:23 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Mihail Nikalayeu <[email protected]> wrote:
> I think I got working POC for deadlock-detector enhancements for
> REPACK (and potentially other).
That looks interesting, I'll check it.
I've also thought about the problem quite a bit this week. I tried to add a
pointer to PGPROC that, like ->waitLock, points to the lock being acquired,
but it's initialized before the actual waiting starts. I adjusted the deadlock
detector to use that pointer too, but it did not work. The problem was
probably that the lock wasn't in the queue during the check.
Finally it occurred to me that a new field can be added to the LOCK structure,
indicating that the lock is being upgraded. It enforces some extra deadlock
checks by other processes, so that the upgrading process does not have to care
about deadlock detection at all. More info in the commit message.
It should handle all the cases in your tests, however a new injection point
would be needed. (Not added yet.)
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-18 22:46 ` Mihail Nikalayeu <[email protected]>
2026-04-20 17:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-18 22:46 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello, Antonin!
I've briefly looked at your patch. As I understand it, it cancels the
other process only when REPACK actually tries to upgrade the lock. In
that sense, its approach is close to my variant at [0].
AFAIU, Andres's concern is that the "victim" should be cancelled
sooner, rather than waiting until REPACK actually attempts the
upgrade. I was trying to solve it by
[1].
-------------------------------
There are some comments on [1]:
Some details related to POC from previous letter (once I re-read that
in the morning I relized it is not so easy to understand, sorry).
So, goal of patch to:
* make sure REPACK will survive deadlock which may caused by lock upgrade
* reject some other backend with error early than deadlock really
occur - to avoid pointless waiting for other commands
* implement it at deadlock detector level to correctly handle
different 2+ backend-involved scenarious
To achive it the first idea is to add some kind of "future lock"
(FutureWaitLock). It may declare an intention to acquire a lock of a
certain level as high-priority in the future.
Once deadlock detector starts to walking graph it treat that intention
like an actual "waiting".
That way deadlock detector looks for one more step in the future -
moment of actual acquiring the "future" lock - and if it ends with
cycle - reject waiting backend ("future deadlock detected").
Looks like best place to put that FutureWaitLock is PGPROC itself.
Few moments to consider:
* it is not allowed to declare a future lock if backend already holds
some kind of lock for the same tag (this may cause a race condition).
* so, REPACK first declares future Access Excluvie and only after it
Share Update Exclusive
* declared future lock should be >= SUE
So far everything looks good.
In case of that scenario:
S1: BEGIN; SELECT * FROM t;
S2: REPACK (CONCURRENTLY) t;
S1: LOCK TABLE t in ACCESS EXCLUSIVE MODE <--- "future deadlock detected"
Deadlock detector sees:
S1 ----> S2 (waiting for AE conflicting with SUE)
S2 ----> S1 (future wait for AE conflict with current Access Share)
But there is a tricky case related to SUE:
S1: BEGIN; SELECT * FROM t;
S2: REPACK (CONCURRENTLY) t;
S1: LOCK TABLE t in SHARE UPDATE EXCLUSIVE MODE;
In that case we have almost the same deadlock scenario - but that is
not visible to deadlock detector.
It happens because SUE does not force all locks taken on 't' to be
transfered using FastPathTransferRelationLocks into the main table
(SUE is does not ConflictsWithRelationFastPath).
Because of it S2 -> S1 edge is not visible by deadlock detector
(Access Share is held using fast-path).
To deal with it we may force any relation with FutureWaitLock to
through slow-path locking - but I don't think it is acceptable.
Instead next approach is proposed:
* deadlock detector checks if any "future" locks are present in the
system (counter in shared memory)
* if so - it iterates over all PROCs to collect relations which are
"future locked"
* for each such relations - FastPathTransferRelationLocks called and
slow-path is forced (FastPathStrongRelationLocks->count)
* deadlock detector start looking for cycles
* once ready - FastPathStrongRelationLocks->count is decremented to
allow fast-path
That way performance degradation happens only during deadlock detector
processing and only if some future locks present.
Due tue LW ordering we need to use some tricks to avoid LW-level
deadlocks (using some kind of retry logic, but that is more explained
in the patch).
[0]: https://www.postgresql.org/message-id/flat/CADzfLwURKVNQ%2B%2BDpi7bjoGfj-8pchDQEVex3eWBx0NCYn6TbDQ%4....
[1}] postgresql.org/message-id/flat/CADzfLwU8Qw6LXFHO7Tbjc-O7o%2BtM26jdnOJBWqYLu61rf7bO%2Bg%40mail.gmail.com#1e96f8882363afb2fc53c2f08346f527
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-18 22:46 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-20 17:44 ` Antonin Houska <[email protected]>
2026-04-23 11:43 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-20 17:44 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Mihail Nikalayeu <[email protected]> wrote:
> I've briefly looked at your patch. As I understand it, it cancels the other
> process only when REPACK actually tries to upgrade the lock. ...
>
> AFAIU, Andres's concern is that the "victim" should be cancelled
> sooner, rather than waiting until REPACK actually attempts the
> upgrade.
I thought the point is that the deadlock should be resolved in a controlled
way, i.e. w/o relying on deadlock timeout. Once both processes sleep, the
decision which one should be kicked off is effectively random. In other words,
the actual deadlock IMO starts exactly at the moment both processes end up
sleeping. But I may be wrong.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-18 22:46 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-20 17:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-23 11:43 ` Mihail Nikalayeu <[email protected]>
2026-04-26 13:34 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-23 11:43 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Mon, Apr 20, 2026 at 7:44 PM Antonin Houska <[email protected]> wrote:
> When another process tries to get the lock after that, it checks this field,
> and if it's set, it checks for deadlocks even if deadlock timeout hasn't
> expired yet. If the process is already in the lock's queue and sleeping, the
> lock upgrading process wakes it up so it checks the flag immediately.
In any way there is no sense in waking up other backends just to force
them to cancel themselves.
Better to go in the [0] way - and just cancel another backend if we
are repack and found cycle in the deadlock detector.
We currently have all required infrastructure, even as the comment
described the possibility [1]:
> * Get this process out of wait state. (Note: we could do this more
> * efficiently by relying on lockAwaited, but use this coding to
> * preserve the flexibility to kill some other transaction than the
> * one detecting the deadlock.)
At the same time version [2] (with FutureWaitLock, explained in more
detail in [3]) is correct and cancels other backends without pointless
waiting, but it feels too complicated due to the complexity of
supporting the FastLock path.
Also, here's one simple idea inspired by your version.
What about adding a new field "do not try to upgrade that lock" to the
LOCK structure? If some backends try to (it has some lockmode and
tries to upgrade it) and it does not has flag 'deadlock_protected'
from [0] - just fail fast with "future deadlock detected".
That way [0] ensures correctness and "do not try to upgrade that lock"
just optimization to fail quickly for the most common scenarios.
Some 2+ backend loops might still wait a long time to be cancelled
(which is solved by [2], but complicated) - but I think this is a
super rare case we can ignore (because repack is still protected from
deadlock).
Mikhail.
[0]: https://www.postgresql.org/message-id/flat/CADzfLwURKVNQ%2B%2BDpi7bjoGfj-8pchDQEVex3eWBx0NCYn6TbDQ%4....
[1]: https://github.com/postgres/postgres/blob/master/src/backend/storage/lmgr/proc.c#L1870-L1873
[2]: https://www.postgresql.org/message-id/flat/CADzfLwUSnGnkfLwCWHQ%3DVVuAY1YTo%2B0Lr7pb%2BOPWUZbcYKSRUw...
[3]: https://www.postgresql.org/message-id/flat/CADzfLwVf-3mjMwSTOcj9djNzGd-UjBOYbFjxgXRhtKuH_4rajA%40mai...
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-18 22:46 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-20 17:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-23 11:43 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-26 13:34 ` Mihail Nikalayeu <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-26 13:34 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
I think I have a good enough approach now (at least balancing
complexity and outcome).
Patch (and commit message) is quite explanatory, but in a few words:
- add 'upgradeIntent' to PROCLOCK (set by REPACK)
- check that in the deadlock detector. If the backend finds the cycle
and is part of it, but because it's upgrading an already announced
lock, it cancels another backend instead of itself.
- use that in the fast path of simple deadlock detection to avoid
pointless waiting (for the easy case involving two backends)
It doesn't cover all scenarios (explained in patch details) but for
majority of realistic scenarios - yes.
It may be extended to cover all of them, but I'm not sure it's worth
the additional complexity.
Best regards,
Mikhail.
Attachments:
[application/octet-stream] nocfbot-v2-0001-Protect-concurrent-repack-lock-upgrades.patch (53.1K, 2-nocfbot-v2-0001-Protect-concurrent-repack-lock-upgrades.patch)
download | inline diff:
From 0717db080b079d0242fc0c3826f1284d793911fd Mon Sep 17 00:00:00 2001
From: Mikhail Nikalayeu <[email protected]>
Date: Sun, 26 Apr 2026 11:50:34 +0200
Subject: [PATCH v2] Protect concurrent repack lock upgrades
REPACK CONCURRENTLY loses significant work when its final ShareUpdateExclusiveLock to AccessExclusiveLock upgrade is canceled by a deadlock. This commit introduces "protected lock upgrades", allowing a backend to atomically record a future upgrade intention (upgradeIntent) on its PROCLOCK via LockAcquireExtended. When a deadlock cycle involves a protected backend's announced upgrade, the detector preempts a blocking waiter rather than the protected upgrader.
Preemption triggers via two paths. JoinWaitQueue fast-fails incoming requests if a holder's upgradeIntent and current modes guarantee a future cycle. Alternatively, if DeadLockCheck evaluates a cycle where the waiter is a protected upgrader, it selects a blocking waiter as the cancellation victim.
---
src/backend/catalog/namespace.c | 32 +++-
src/backend/commands/repack.c | 29 ++-
src/backend/storage/lmgr/README | 26 +++
src/backend/storage/lmgr/deadlock.c | 90 ++++++++-
src/backend/storage/lmgr/lmgr.c | 49 +++--
src/backend/storage/lmgr/lock.c | 154 +++++++++++++--
src/backend/storage/lmgr/proc.c | 134 ++++++++++---
src/include/catalog/namespace.h | 7 +
src/include/storage/lmgr.h | 3 +
src/include/storage/lock.h | 19 +-
src/include/storage/proc.h | 7 -
.../injection_points/expected/repack.out | 181 +++++++++++++++++-
.../injection_points/specs/repack.spec | 108 +++++++++++
13 files changed, 757 insertions(+), 82 deletions(-)
diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c
index 56b87d878e8..31a958198e0 100644
--- a/src/backend/catalog/namespace.c
+++ b/src/backend/catalog/namespace.c
@@ -415,8 +415,18 @@ spcache_insert(const char *searchPath, Oid roleid)
}
}
+/* Wrapper preserving the historical signature (no upgrade intent). */
+Oid
+RangeVarGetRelidExtended(const RangeVar *relation, LOCKMODE lockmode,
+ uint32 flags,
+ RangeVarGetRelidCallback callback, void *callback_arg)
+{
+ return RangeVarGetRelidWithUpgradeIntent(relation, lockmode, NoLock,
+ flags, callback, callback_arg);
+}
+
/*
- * RangeVarGetRelidExtended
+ * RangeVarGetRelidWithUpgradeIntent
* Given a RangeVar describing an existing relation,
* select the proper namespace and look up the relation OID.
*
@@ -435,13 +445,17 @@ spcache_insert(const char *searchPath, Oid roleid)
* return value of InvalidOid could either mean the relation is missing or it
* could not be locked.
*
+ * If upgradeMode is not NoLock, an upgrade-intent announcement is installed
+ * atomically with the grant; see LockAcquireExtended.
+ *
* Callback allows caller to check permissions or acquire additional locks
* prior to grabbing the relation lock.
*/
Oid
-RangeVarGetRelidExtended(const RangeVar *relation, LOCKMODE lockmode,
- uint32 flags,
- RangeVarGetRelidCallback callback, void *callback_arg)
+RangeVarGetRelidWithUpgradeIntent(const RangeVar *relation, LOCKMODE lockmode,
+ LOCKMODE upgradeMode, uint32 flags,
+ RangeVarGetRelidCallback callback,
+ void *callback_arg)
{
uint64 inval_count;
Oid relId;
@@ -451,6 +465,9 @@ RangeVarGetRelidExtended(const RangeVar *relation, LOCKMODE lockmode,
/* verify that flags do no conflict */
Assert(!((flags & RVR_NOWAIT) && (flags & RVR_SKIP_LOCKED)));
+ /* Upgrade intent requires the blocking acquisition path. */
+ Assert(upgradeMode == NoLock ||
+ !(flags & (RVR_NOWAIT | RVR_SKIP_LOCKED)));
/*
* We check the catalog name and then ignore it.
@@ -590,7 +607,12 @@ RangeVarGetRelidExtended(const RangeVar *relation, LOCKMODE lockmode,
if (!OidIsValid(relId))
AcceptInvalidationMessages();
else if (!(flags & (RVR_NOWAIT | RVR_SKIP_LOCKED)))
- LockRelationOid(relId, lockmode);
+ {
+ if (upgradeMode != NoLock)
+ LockRelationOidWithUpgradeIntent(relId, lockmode, upgradeMode);
+ else
+ LockRelationOid(relId, lockmode);
+ }
else if (!ConditionalLockRelationOid(relId, lockmode))
{
int elevel = (flags & RVR_SKIP_LOCKED) ? DEBUG1 : ERROR;
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index bafdca80810..c63d2107ef2 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -2322,12 +2322,25 @@ process_single_relation(RepackStmt *stmt, LOCKMODE lockmode, bool isTopLevel,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("ANALYZE option must be specified when a column list is provided"));
- /* Find, lock, and check permissions on the table. */
- tableOid = RangeVarGetRelidExtended(stmt->relation->relation,
- lockmode,
- 0,
- RangeVarCallbackMaintainsTable,
- NULL);
+ /*
+ * Find, lock, and check permissions on the table.
+ *
+ * For CONCURRENTLY, announce the future AEL upgrade so a conflicting
+ * holder is preempted instead of stalling us through the copy phase.
+ */
+ if ((params->options & CLUOPT_CONCURRENT) != 0)
+ tableOid = RangeVarGetRelidWithUpgradeIntent(stmt->relation->relation,
+ lockmode,
+ AccessExclusiveLock,
+ 0,
+ RangeVarCallbackMaintainsTable,
+ NULL);
+ else
+ tableOid = RangeVarGetRelidExtended(stmt->relation->relation,
+ lockmode,
+ 0,
+ RangeVarCallbackMaintainsTable,
+ NULL);
rel = table_open(tableOid, NoLock);
/*
@@ -3055,6 +3068,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
/*
* Acquire AccessExclusiveLock on the table, its TOAST relation (if there
* is one), all its indexes, so that we can swap the files.
+ *
+ * The upgrade intent announced at initial SUEL acquisition makes the
+ * detector preempt a blocker on a deadlock here, preserving the work
+ * already done.
*/
LockRelationOid(old_table_oid, AccessExclusiveLock);
diff --git a/src/backend/storage/lmgr/README b/src/backend/storage/lmgr/README
index 45de0fd2bd6..ced7d285279 100644
--- a/src/backend/storage/lmgr/README
+++ b/src/backend/storage/lmgr/README
@@ -586,6 +586,32 @@ The caller can then send a cancellation signal. This implements the
principle that autovacuum has a low locking priority (eg it must not block
DDL on the table).
+Protected Lock Upgrades
+-----------------------
+
+A backend can announce at initial-lock acquisition that it intends to upgrade
+to a stronger mode later (PROCLOCK->upgradeIntent), installed atomically with
+the grant via LockAcquireExtended. REPACK CONCURRENTLY uses this so that a
+deadlock on its final SUEL->AEL swap cancels a blocker rather than throwing
+away the copy/rebuild phase.
+
+When a cycle involving the protected backend is detected, a blocking waiter is
+canceled instead of the protected backend. This can happen either in
+JoinWaitQueue's fast-fail check or in the full DeadLockCheck() during the
+announced upgrade wait. The fast-fail check is only a cheap early test for
+cycles already visible in the main lock table. It does not force transfer of
+unrelated fast-path relation locks, so it may miss cycles whose edges are still
+hidden in backend-local fast-path state. Such cycles become visible later when a
+conflicting strong lock request transfers the relevant fast-path locks.
+The full deadlock detector handles that case.
+
+Other protected upgraders and anti-wraparound autovacuums are skipped as
+victims. If no acceptable victim exists, the protected backend is canceled as
+usual.
+
+The victim observes preemption via PROC_WAIT_STATUS_PREEMPTED on its
+waitStatus, which LockAcquireExtended turns into a deadlock ereport.
+
Group Locking
-------------
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index b8962d875b6..35143f0ab1e 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -125,6 +125,13 @@ static int maxPossibleConstraints;
static DEADLOCK_INFO *deadlockDetails;
static int nDeadlockDetails;
+/*
+ * Parallel to deadlockDetails[]: live PGPROCs forming each edge of the
+ * detected cycle. Valid only while all partition locks are held. Used by
+ * DeadLockCheck to pick a victim under the protected-upgrade rule.
+ */
+static PGPROC **deadlockProcs;
+
/* PGPROC pointer of any blocking autovacuum worker found */
static PGPROC *blocking_autovacuum_proc = NULL;
@@ -148,11 +155,12 @@ InitDeadLockChecking(void)
oldcxt = MemoryContextSwitchTo(TopMemoryContext);
/*
- * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
- * deadlockDetails[].
+ * FindLockCycle needs at most MaxBackends entries in visitedProcs[],
+ * deadlockDetails[], and deadlockProcs[].
*/
visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+ deadlockProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
/*
* TopoSort needs to consider at most MaxBackends wait-queue entries, and
@@ -219,10 +227,14 @@ InitDeadLockChecking(void)
DeadLockState
DeadLockCheck(PGPROC *proc)
{
+ bool victims_canceled = false;
+
+retry:
/* Initialize to "no constraints" */
nCurConstraints = 0;
nPossibleConstraints = 0;
nWaitOrders = 0;
+ nDeadlockDetails = 0;
/* Initialize to not blocked by an autovacuum worker */
blocking_autovacuum_proc = NULL;
@@ -242,6 +254,72 @@ DeadLockCheck(PGPROC *proc)
if (!FindLockCycle(proc, possibleConstraints, &nSoftEdges))
elog(FATAL, "deadlock seems to have disappeared");
+ /*
+ * Protected-upgrade resolution: if our wait is the announced upgrade
+ * for our proclock, cancel a blocker instead of MyProc and re-run the
+ * full check (not just FindLockCycle) so soft cycles still get a
+ * chance at queue rearrangement. Partition locks are held throughout,
+ * so the loop is bounded by MaxBackends.
+ *
+ * deadlockProcs[0] is always MyProc; start at the direct blocker.
+ * Skip other protected upgraders (don't preempt them) and
+ * anti-wraparound autovacuums (canceling them is worse than losing
+ * our work). If no acceptable victim exists, fall through to
+ * DS_HARD_DEADLOCK.
+ */
+ if (MyProc->waitProcLock != NULL &&
+ MyProc->waitProcLock->upgradeIntent == MyProc->waitLockMode)
+ {
+ PGPROC *victim = NULL;
+
+ for (int i = 1; i < nDeadlockDetails; i++)
+ {
+ PGPROC *w = deadlockProcs[i];
+
+ if (w == NULL || w == MyProc)
+ continue;
+
+ /* Don't preempt another protected upgrader. */
+ if (w->waitProcLock != NULL &&
+ w->waitProcLock->upgradeIntent == w->waitLockMode)
+ continue;
+
+ /*
+ * PROC_IS_AUTOVACUUM is set once at startup, safe lockless.
+ * Read PROC_VACUUM_FOR_WRAPAROUND only if we can grab
+ * ProcArrayLock conditionally; otherwise skip this candidate.
+ */
+ if (w->statusFlags & PROC_IS_AUTOVACUUM)
+ {
+ uint8 statusFlags;
+
+ if (!LWLockConditionalAcquire(ProcArrayLock, LW_SHARED))
+ continue;
+
+ statusFlags = ProcGlobal->statusFlags[w->pgxactoff];
+ LWLockRelease(ProcArrayLock);
+
+ if (statusFlags & PROC_VACUUM_FOR_WRAPAROUND)
+ continue;
+ }
+
+ Assert(w->waitLock != NULL);
+ Assert(!dlist_node_is_detached(&w->waitLink));
+ victim = w;
+ break;
+ }
+
+ if (victim != NULL)
+ {
+ RemoveFromWaitQueue(victim,
+ LockTagHashCode(&(victim->waitLock->tag)),
+ PROC_WAIT_STATUS_PREEMPTED);
+ SetLatch(&victim->procLatch);
+ victims_canceled = true;
+ goto retry;
+ }
+ }
+
return DS_HARD_DEADLOCK; /* cannot find a non-deadlocked state */
}
@@ -273,7 +351,9 @@ DeadLockCheck(PGPROC *proc)
}
/* Return code tells caller if we had to escape a deadlock or not */
- if (nWaitOrders > 0)
+ if (victims_canceled)
+ return DS_PREEMPT_DEADLOCK;
+ else if (nWaitOrders > 0)
return DS_SOFT_DEADLOCK;
else if (blocking_autovacuum_proc != NULL)
return DS_BLOCKED_BY_AUTOVACUUM;
@@ -590,6 +670,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
return true;
}
@@ -679,6 +760,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
/*
* Add this edge to the list of soft edges in the cycle
@@ -753,6 +835,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
info->locktag = lock->tag;
info->lockmode = checkProc->waitLockMode;
info->pid = checkProc->pid;
+ deadlockProcs[depth] = checkProc;
/*
* Add this edge to the list of soft edges in the cycle
@@ -1159,4 +1242,5 @@ RememberSimpleDeadLock(PGPROC *proc1,
info->lockmode = proc2->waitLockMode;
info->pid = proc2->pid;
nDeadlockDetails = 2;
+ deadlockProcs[0] = deadlockProcs[1] = NULL;
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index 2ccf7237fee..10b0e0a5b9a 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -105,6 +105,19 @@ SetLocktagRelationOid(LOCKTAG *tag, Oid relid)
*/
void
LockRelationOid(Oid relid, LOCKMODE lockmode)
+{
+ LockRelationOidWithUpgradeIntent(relid, lockmode, NoLock);
+}
+
+/*
+ * LockRelationOidWithUpgradeIntent
+ *
+ * Lock and atomically announce a future upgrade to `upgradeMode`. See
+ * LockAcquireExtended for semantics.
+ */
+void
+LockRelationOidWithUpgradeIntent(Oid relid, LOCKMODE lockmode,
+ LOCKMODE upgradeMode)
{
LOCKTAG tag;
LOCALLOCK *locallock;
@@ -112,8 +125,8 @@ LockRelationOid(Oid relid, LOCKMODE lockmode)
SetLocktagRelationOid(&tag, relid);
- res = LockAcquireExtended(&tag, lockmode, false, false, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, upgradeMode, false, false, true,
+ &locallock, false);
/*
* Now that we have the lock, check for invalidation messages, so that we
@@ -156,8 +169,8 @@ ConditionalLockRelationOid(Oid relid, LOCKMODE lockmode)
SetLocktagRelationOid(&tag, relid);
- res = LockAcquireExtended(&tag, lockmode, false, true, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, true, true,
+ &locallock, false);
if (res == LOCKACQUIRE_NOT_AVAIL)
return false;
@@ -190,8 +203,8 @@ LockRelationId(LockRelId *relid, LOCKMODE lockmode)
SET_LOCKTAG_RELATION(tag, relid->dbId, relid->relId);
- res = LockAcquireExtended(&tag, lockmode, false, false, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, false, true,
+ &locallock, false);
/*
* Now that we have the lock, check for invalidation messages; see notes
@@ -253,8 +266,8 @@ LockRelation(Relation relation, LOCKMODE lockmode)
relation->rd_lockInfo.lockRelId.dbId,
relation->rd_lockInfo.lockRelId.relId);
- res = LockAcquireExtended(&tag, lockmode, false, false, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, false, true,
+ &locallock, false);
/*
* Now that we have the lock, check for invalidation messages; see notes
@@ -285,8 +298,8 @@ ConditionalLockRelation(Relation relation, LOCKMODE lockmode)
relation->rd_lockInfo.lockRelId.dbId,
relation->rd_lockInfo.lockRelId.relId);
- res = LockAcquireExtended(&tag, lockmode, false, true, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, true, true,
+ &locallock, false);
if (res == LOCKACQUIRE_NOT_AVAIL)
return false;
@@ -590,8 +603,8 @@ ConditionalLockTuple(Relation relation, const ItemPointerData *tid, LOCKMODE loc
ItemPointerGetBlockNumber(tid),
ItemPointerGetOffsetNumber(tid));
- return (LockAcquireExtended(&tag, lockmode, false, true, true, NULL,
- logLockFailure) != LOCKACQUIRE_NOT_AVAIL);
+ return (LockAcquireExtended(&tag, lockmode, NoLock, false, true, true,
+ NULL, logLockFailure) != LOCKACQUIRE_NOT_AVAIL);
}
/*
@@ -748,8 +761,8 @@ ConditionalXactLockTableWait(TransactionId xid, bool logLockFailure)
SET_LOCKTAG_TRANSACTION(tag, xid);
- if (LockAcquireExtended(&tag, ShareLock, false, true, true, NULL,
- logLockFailure)
+ if (LockAcquireExtended(&tag, ShareLock, NoLock, false, true, true,
+ NULL, logLockFailure)
== LOCKACQUIRE_NOT_AVAIL)
return false;
@@ -1042,8 +1055,8 @@ ConditionalLockDatabaseObject(Oid classid, Oid objid, uint16 objsubid,
objid,
objsubid);
- res = LockAcquireExtended(&tag, lockmode, false, true, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, true, true,
+ &locallock, false);
if (res == LOCKACQUIRE_NOT_AVAIL)
return false;
@@ -1122,8 +1135,8 @@ ConditionalLockSharedObject(Oid classid, Oid objid, uint16 objsubid,
objid,
objsubid);
- res = LockAcquireExtended(&tag, lockmode, false, true, true, &locallock,
- false);
+ res = LockAcquireExtended(&tag, lockmode, NoLock, false, true, true,
+ &locallock, false);
if (res == LOCKACQUIRE_NOT_AVAIL)
return false;
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 8d246ed5a4e..0e666a025ef 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -59,6 +59,9 @@ bool log_lock_failures = false;
#define NLOCKENTS() \
mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+#define LOCKMODE_SELF_CONFLICTS(conflictTab, lockmode) \
+ (((conflictTab)[(lockmode)] & LOCKBIT_ON(lockmode)) != 0)
+
/*
* Data structures defining the semantics of the standard lock methods.
@@ -384,7 +387,7 @@ LOCK_PRINT(const char *where, const LOCK *lock, LOCKMODE type)
elog(LOG,
"%s: lock(%p) id(%u,%u,%u,%u,%u,%u) grantMask(%x) "
"req(%d,%d,%d,%d,%d,%d,%d)=%d "
- "grant(%d,%d,%d,%d,%d,%d,%d)=%d wait(%d) type(%s)",
+ "grant(%d,%d,%d,%d,%d,%d,%d)=%d wait(%d) upgradeIntent(%d) type(%s)",
where, lock,
lock->tag.locktag_field1, lock->tag.locktag_field2,
lock->tag.locktag_field3, lock->tag.locktag_field4,
@@ -397,6 +400,7 @@ LOCK_PRINT(const char *where, const LOCK *lock, LOCKMODE type)
lock->granted[4], lock->granted[5], lock->granted[6],
lock->granted[7], lock->nGranted,
dclist_count(&lock->waitProcs),
+ lock->nUpgradeIntent,
LockMethods[LOCK_LOCKMETHOD(*lock)]->lockModeNames[type]);
}
@@ -406,10 +410,11 @@ PROCLOCK_PRINT(const char *where, const PROCLOCK *proclockP)
{
if (LOCK_DEBUG_ENABLED(&proclockP->tag.myLock->tag))
elog(LOG,
- "%s: proclock(%p) lock(%p) method(%u) proc(%p) hold(%x)",
+ "%s: proclock(%p) lock(%p) method(%u) proc(%p) hold(%x) upgradeIntent(%d)",
where, proclockP, proclockP->tag.myLock,
PROCLOCK_LOCKMETHOD(*(proclockP)),
- proclockP->tag.myProc, (int) proclockP->holdMask);
+ proclockP->tag.myProc, (int) proclockP->holdMask,
+ proclockP->upgradeIntent);
}
#else /* not LOCK_DEBUG */
@@ -427,6 +432,7 @@ static void BeginStrongLockAcquire(LOCALLOCK *locallock, uint32 fasthashcode);
static void FinishStrongLockAcquire(void);
static ProcWaitStatus WaitOnLock(LOCALLOCK *locallock, ResourceOwner owner);
static void waitonlock_error_callback(void *arg);
+pg_noreturn static void ReportPreemptDeadlock(LOCALLOCK *locallock);
static void ReleaseLockIfHeld(LOCALLOCK *locallock, bool sessionLock);
static void LockReassignOwner(LOCALLOCK *locallock, ResourceOwner parent);
static bool UnGrantLock(LOCK *lock, LOCKMODE lockmode,
@@ -808,13 +814,17 @@ LockAcquire(const LOCKTAG *locktag,
bool sessionLock,
bool dontWait)
{
- return LockAcquireExtended(locktag, lockmode, sessionLock, dontWait,
- true, NULL, false);
+ return LockAcquireExtended(locktag, lockmode, NoLock, sessionLock,
+ dontWait, true, NULL, false);
}
/*
* LockAcquireExtended - allows us to specify additional options
*
+ * upgradeIntent is NoLock or a stronger mode the caller will try to acquire
+ * later; when set, it is installed on the PROCLOCK atomically with the grant.
+ * See README "Protected Lock Upgrades".
+ *
* reportMemoryError specifies whether a lock request that fills the lock
* table should generate an ERROR or not. Passing "false" allows the caller
* to attempt to recover from lock-table-full situations, perhaps by forcibly
@@ -832,6 +842,7 @@ LockAcquire(const LOCKTAG *locktag,
LockAcquireResult
LockAcquireExtended(const LOCKTAG *locktag,
LOCKMODE lockmode,
+ LOCKMODE upgradeIntent,
bool sessionLock,
bool dontWait,
bool reportMemoryError,
@@ -851,6 +862,7 @@ LockAcquireExtended(const LOCKTAG *locktag,
bool found_conflict;
ProcWaitStatus waitResult;
bool log_lock = false;
+ bool upgradeIntentSet = false;
if (lockmethodid <= 0 || lockmethodid >= lengthof(LockMethods))
elog(ERROR, "unrecognized lock method: %d", lockmethodid);
@@ -858,6 +870,16 @@ LockAcquireExtended(const LOCKTAG *locktag,
if (lockmode <= 0 || lockmode > lockMethodTable->numLockModes)
elog(ERROR, "unrecognized lock mode: %d", lockmode);
+ /*
+ * Announcement requires a self-conflicting non-fast-path lockmode (the
+ * fast path can't carry one) and a strictly stronger target.
+ */
+ Assert(upgradeIntent == NoLock ||
+ (!EligibleForRelationFastPath(locktag, lockmode) &&
+ LOCKMODE_SELF_CONFLICTS(lockMethodTable->conflictTab, lockmode) &&
+ upgradeIntent > lockmode &&
+ upgradeIntent <= lockMethodTable->numLockModes));
+
if (RecoveryInProgress() && !InRecovery &&
(locktag->locktag_type == LOCKTAG_OBJECT ||
locktag->locktag_type == LOCKTAG_RELATION) &&
@@ -933,9 +955,13 @@ LockAcquireExtended(const LOCKTAG *locktag,
*
* If lockCleared is already set, caller need not worry about absorbing
* sinval messages related to the lock's object.
+ *
+ * Upgrade intent must be installed at the first grant, not on re-entry.
*/
if (locallock->nLocks > 0)
{
+ Assert(upgradeIntent == NoLock);
+
GrantLockLocal(locallock, owner);
if (locallock->lockCleared)
return LOCKACQUIRE_ALREADY_CLEAR;
@@ -1094,6 +1120,19 @@ LockAcquireExtended(const LOCKTAG *locktag,
lock = proclock->tag.myLock;
locallock->lock = lock;
+ if (upgradeIntent != NoLock)
+ {
+ /* It is not allowed to change upgradeIntent mode */
+ Assert(proclock->upgradeIntent == NoLock ||
+ proclock->upgradeIntent == upgradeIntent);
+ upgradeIntentSet = (proclock->upgradeIntent == NoLock);
+ if (upgradeIntentSet)
+ {
+ proclock->upgradeIntent = upgradeIntent;
+ lock->nUpgradeIntent++;
+ }
+ }
+
/*
* If lock requested conflicts with locks requested by waiters, must join
* wait queue. Otherwise, check for conflict with already-held locks.
@@ -1121,18 +1160,26 @@ LockAcquireExtended(const LOCKTAG *locktag,
waitResult = JoinWaitQueue(locallock, lockMethodTable, dontWait);
}
- if (waitResult == PROC_WAIT_STATUS_ERROR)
+ if (waitResult == PROC_WAIT_STATUS_ERROR ||
+ waitResult == PROC_WAIT_STATUS_PREEMPTED)
{
/*
- * We're not getting the lock because a deadlock was detected already
- * while trying to join the wait queue, or because we would have to
- * wait but the caller requested no blocking.
- *
+ * Not getting the lock: deadlock detected while joining the queue,
+ * preempted by another backend's protected upgrade, or dontWait.
* Undo the changes to shared entries before releasing the partition
* lock.
*/
AbortStrongLockAcquire();
+ /* If we just installed upgrade intent but never got the lock, undo it. */
+ if (upgradeIntentSet &&
+ (proclock->holdMask & LOCKBIT_ON(lockmode)) == 0)
+ {
+ proclock->upgradeIntent = NoLock;
+ Assert(lock->nUpgradeIntent > 0);
+ lock->nUpgradeIntent--;
+ }
+
if (proclock->holdMask == 0)
{
uint32 proclock_hashcode;
@@ -1208,6 +1255,8 @@ LockAcquireExtended(const LOCKTAG *locktag,
*locallockp = NULL;
return LOCKACQUIRE_NOT_AVAIL;
}
+ else if (waitResult == PROC_WAIT_STATUS_PREEMPTED)
+ ReportPreemptDeadlock(locallock);
else
{
DeadLockReport();
@@ -1236,14 +1285,15 @@ LockAcquireExtended(const LOCKTAG *locktag,
if (waitResult == PROC_WAIT_STATUS_ERROR)
{
- /*
- * We failed as a result of a deadlock, see CheckDeadLock(). Quit
- * now.
- */
Assert(!dontWait);
DeadLockReport();
/* DeadLockReport() will not return */
}
+ else if (waitResult == PROC_WAIT_STATUS_PREEMPTED)
+ {
+ Assert(!dontWait);
+ ReportPreemptDeadlock(locallock);
+ }
}
else
LWLockRelease(partitionLock);
@@ -1319,6 +1369,7 @@ SetupLockInTable(LockMethod lockMethodTable, PGPROC *proc,
dclist_init(&lock->waitProcs);
lock->nRequested = 0;
lock->nGranted = 0;
+ lock->nUpgradeIntent = 0;
MemSet(lock->requested, 0, sizeof(int) * MAX_LOCKMODES);
MemSet(lock->granted, 0, sizeof(int) * MAX_LOCKMODES);
LOCK_PRINT("LockAcquire: new", lock, lockmode);
@@ -1390,6 +1441,7 @@ SetupLockInTable(LockMethod lockMethodTable, PGPROC *proc,
proc->lockGroupLeader : proc;
proclock->holdMask = 0;
proclock->releaseMask = 0;
+ proclock->upgradeIntent = NoLock;
/* Add proclock to appropriate lists */
dlist_push_tail(&lock->procLocks, &proclock->lockLink);
dlist_push_tail(&proc->myProcLocks[partition], &proclock->procLink);
@@ -1413,9 +1465,13 @@ SetupLockInTable(LockMethod lockMethodTable, PGPROC *proc,
* about user-level coding practices that are in fact safe in context.
* It can be enabled to help find system-level problems.
*
+ * Skip for an announced upgrade: the detector handles those cycles
+ * by canceling a blocker.
+ *
* XXX Doing numeric comparison on the lockmodes is a hack; it'd be
* better to use a table. For now, though, this works.
*/
+ if (proclock->upgradeIntent != lockmode)
{
int i;
@@ -1671,6 +1727,19 @@ GrantLock(LOCK *lock, PROCLOCK *proclock, LOCKMODE lockmode)
if (lock->granted[lockmode] == lock->requested[lockmode])
lock->waitMask &= LOCKBIT_OFF(lockmode);
proclock->holdMask |= LOCKBIT_ON(lockmode);
+
+ /* The announced target was just granted; clear the announcement. */
+ if (proclock->upgradeIntent != NoLock)
+ {
+ Assert(proclock->upgradeIntent >= lockmode);
+ if (proclock->upgradeIntent == lockmode)
+ {
+ proclock->upgradeIntent = NoLock;
+ Assert(lock->nUpgradeIntent > 0);
+ lock->nUpgradeIntent--;
+ }
+ }
+
LOCK_PRINT("GrantLock", lock, lockmode);
Assert((lock->nGranted > 0) && (lock->granted[lockmode] > 0));
Assert(lock->nGranted <= lock->nRequested);
@@ -1727,6 +1796,16 @@ UnGrantLock(LOCK *lock, LOCKMODE lockmode,
* Now fix the per-proclock state.
*/
proclock->holdMask &= LOCKBIT_OFF(lockmode);
+
+ /* Releasing the last held mode also retires any pending announcement. */
+ if (proclock->upgradeIntent != NoLock &&
+ proclock->holdMask == 0)
+ {
+ proclock->upgradeIntent = NoLock;
+ Assert(lock->nUpgradeIntent > 0);
+ lock->nUpgradeIntent--;
+ }
+
PROCLOCK_PRINT("UnGrantLock: updated", proclock);
return wakeupNeeded;
@@ -1883,6 +1962,29 @@ AbortStrongLockAcquire(void)
SpinLockRelease(&FastPathStrongRelationLocks->mutex);
}
+/*
+ * ereport a deadlock for a protected-upgrade preemption. Use log_lock_waits
+ * for the cycle details from the protected backend's side.
+ */
+pg_noreturn static void
+ReportPreemptDeadlock(LOCALLOCK *locallock)
+{
+ StringInfoData buf;
+ const char *modename;
+
+ initStringInfo(&buf);
+ DescribeLockTag(&buf, &locallock->tag.lock);
+ modename = GetLockmodeName(locallock->tag.lock.locktag_lockmethodid,
+ locallock->tag.mode);
+
+ pgstat_report_deadlock();
+ ereport(ERROR,
+ (errcode(ERRCODE_T_R_DEADLOCK_DETECTED),
+ errmsg("deadlock detected"),
+ errdetail("Process %d could not acquire %s on %s because another backend holds a conflicting lock and has announced a future upgrade that would form a deadlock cycle.",
+ MyProcPid, modename, buf.data)));
+}
+
/*
* GrantAwaitedLock -- call GrantLockLocal for the lock we are doing
* WaitOnLock on.
@@ -2042,8 +2144,10 @@ waitonlock_error_callback(void *arg)
/*
* Remove a proc from the wait-queue it is on (caller must know it is on one).
- * This is only used when the proc has failed to get the lock, so we set its
- * waitStatus to PROC_WAIT_STATUS_ERROR.
+ * Used when the proc has failed to get the lock; the caller passes the
+ * waitStatus to publish (PROC_WAIT_STATUS_ERROR for ordinary failure or
+ * PROC_WAIT_STATUS_PREEMPTED when the wait was canceled to break a cycle
+ * with a protected upgrade).
*
* Appropriate partition lock must be held by caller. Also, caller is
* responsible for signaling the proc if needed.
@@ -2051,7 +2155,7 @@ waitonlock_error_callback(void *arg)
* NB: this does not clean up any locallock object that may exist for the lock.
*/
void
-RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
+RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode, ProcWaitStatus newStatus)
{
LOCK *waitLock = proc->waitLock;
PROCLOCK *proclock = proc->waitProcLock;
@@ -2060,6 +2164,8 @@ RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
/* Make sure proc is waiting */
Assert(proc->waitStatus == PROC_WAIT_STATUS_WAITING);
+ Assert(newStatus == PROC_WAIT_STATUS_ERROR ||
+ newStatus == PROC_WAIT_STATUS_PREEMPTED);
Assert(!dlist_node_is_detached(&proc->waitLink));
Assert(waitLock);
Assert(!dclist_is_empty(&waitLock->waitProcs));
@@ -2078,10 +2184,18 @@ RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode)
if (waitLock->granted[lockmode] == waitLock->requested[lockmode])
waitLock->waitMask &= LOCKBIT_OFF(lockmode);
+ /* Wait canceled => the announcement (if any) no longer applies. */
+ if (proclock->upgradeIntent != NoLock)
+ {
+ proclock->upgradeIntent = NoLock;
+ Assert(waitLock->nUpgradeIntent > 0);
+ waitLock->nUpgradeIntent--;
+ }
+
/* Clean up the proc's own state, and pass it the ok/fail signal */
proc->waitLock = NULL;
proc->waitProcLock = NULL;
- proc->waitStatus = PROC_WAIT_STATUS_ERROR;
+ proc->waitStatus = newStatus;
/*
* Delete the proclock immediately if it represents no already-held locks.
@@ -3697,6 +3811,8 @@ PostPrepare_Locks(FullTransactionId fxid)
Assert(lock->nGranted >= 0);
Assert(lock->nGranted <= lock->nRequested);
Assert((proclock->holdMask & ~lock->grantMask) == 0);
+ /* upgradeIntent is not supported for PREPARE TRANSACTION. */
+ Assert(proclock->upgradeIntent == NoLock);
/* Ignore it if nothing to release (must be a session lock) */
if (proclock->releaseMask == 0)
@@ -4397,6 +4513,7 @@ lock_twophase_recover(FullTransactionId fxid, uint16 info,
dclist_init(&lock->waitProcs);
lock->nRequested = 0;
lock->nGranted = 0;
+ lock->nUpgradeIntent = 0;
MemSet(lock->requested, 0, sizeof(int) * MAX_LOCKMODES);
MemSet(lock->granted, 0, sizeof(int) * MAX_LOCKMODES);
LOCK_PRINT("lock_twophase_recover: new", lock, lockmode);
@@ -4460,6 +4577,7 @@ lock_twophase_recover(FullTransactionId fxid, uint16 info,
proclock->groupLeader = proc;
proclock->holdMask = 0;
proclock->releaseMask = 0;
+ proclock->upgradeIntent = NoLock;
/* Add proclock to appropriate lists */
dlist_push_tail(&lock->procLocks, &proclock->lockLink);
dlist_push_tail(&proc->myProcLocks[partition],
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1ac25068d62..2ce7073d9d2 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -854,7 +854,8 @@ LockErrorCleanup(void)
if (!dlist_node_is_detached(&MyProc->waitLink))
{
/* We could not have been granted the lock yet */
- RemoveFromWaitQueue(MyProc, lockAwaited->hashcode);
+ RemoveFromWaitQueue(MyProc, lockAwaited->hashcode,
+ PROC_WAIT_STATUS_ERROR);
}
else
{
@@ -1135,10 +1136,11 @@ AuxiliaryPidGetProc(int pid)
*
* Result is one of the following:
*
- * PROC_WAIT_STATUS_OK - lock was immediately granted
- * PROC_WAIT_STATUS_WAITING - joined the wait queue; call ProcSleep()
- * PROC_WAIT_STATUS_ERROR - immediate deadlock was detected, or would
- * need to wait and dontWait == true
+ * PROC_WAIT_STATUS_OK - lock immediately granted
+ * PROC_WAIT_STATUS_WAITING - joined the wait queue; call ProcSleep()
+ * PROC_WAIT_STATUS_ERROR - immediate deadlock or dontWait failure
+ * PROC_WAIT_STATUS_PREEMPTED - cycle proven against an announced upgrade;
+ * caller must raise a deadlock error
*
* NOTES: The process queue is now a priority queue for locking.
*/
@@ -1239,9 +1241,15 @@ JoinWaitQueue(LOCALLOCK *locallock, LockMethod lockMethodTable, bool dontWait)
* can't do that until we are *on* the wait queue. So, set
* a flag to check below, and break out of loop. Also,
* record deadlock info for later message.
+ *
+ * If this wait is our announced upgrade, skip the early
+ * exit so the full detector can preempt a blocker.
*/
- RememberSimpleDeadLock(MyProc, lockmode, lock, proc);
- early_deadlock = true;
+ if (proclock->upgradeIntent != lockmode)
+ {
+ RememberSimpleDeadLock(MyProc, lockmode, lock, proc);
+ early_deadlock = true;
+ }
break;
}
/* I must go before this waiter. Check special case. */
@@ -1270,6 +1278,75 @@ JoinWaitQueue(LOCALLOCK *locallock, LockMethod lockMethodTable, bool dontWait)
if (early_deadlock)
return PROC_WAIT_STATUS_ERROR;
+ /*
+ * Fast-fail: if some holder's announced upgrade conflicts with a mode we
+ * already hold, and its current holdMask conflicts with our request, a
+ * future cycle is inevitable. Don't wait deadlock_timeout for the same
+ * conclusion. Skipped when we are the announced upgrader, when we hold
+ * nothing here, or when no holder has any announcement.
+ *
+ * This is only a fast-fail check for cycles already visible in the main
+ * lock table. Fast-path locks can hide edges. For example, while waiting
+ * for ShareUpdateExclusiveLock, this backend may still hold weaker fast-path
+ * locks on the same relation, since ShareUpdateExclusiveLock does not itself
+ * force fast-path transfer. Such an edge becomes visible only later, e.g.
+ * when another backend's AccessExclusiveLock upgrade transfers fast-path
+ * locks. The full deadlock detector handles that case.
+ *
+ * We could cheaply include modes this backend holds locally, via
+ * LockMethodLocalHash. That catches the direct case where this backend holds
+ * a weak fast-path lock and waits behind an announced stronger upgrade, but
+ * it cannot see hidden fast-path edges owned by third backends in longer
+ * cycles.
+ *
+ * Making upgrade intent visible to the fast-path mechanism would catch more
+ * cases, e.g. by treating upgradeIntent as conflicting for
+ * ConflictsWithRelationFastPath() and keeping FastPathStrongRelationLocks
+ * bumped until the upgrade is consumed. However, that would pessimize the
+ * whole concurrent operation: ordinary weak relation locks would stop using
+ * the fast path and contend in the main lock table.
+ *
+ * Transferring fast-path locks from DeadLockCheck() only when needed would
+ * avoid that penalty, but is lock-ordering-sensitive: DeadLockCheck() already
+ * holds all lock partition LWLocks, while fast-path transfer needs
+ * fpInfoLocks and may acquire partition LWLocks in the normal order. A safe
+ * design would need probe/retry logic, making it worthwhile only if hidden
+ * longer fast-path cycles must be preempted early.
+ */
+ if (proclock->upgradeIntent != lockmode && myHeldLocks != 0 &&
+ lock->nUpgradeIntent > 0)
+ {
+ dlist_iter iter;
+
+ dlist_foreach(iter, &lock->procLocks)
+ {
+ PROCLOCK *holder = dlist_container(PROCLOCK, lockLink, iter.cur);
+
+ if (holder->upgradeIntent == NoLock)
+ continue;
+
+ /* Skip self (we're in lock->procLocks too). */
+ if (holder == proclock)
+ continue;
+
+ /* Same lock group: not a cycle. */
+ if (leader != NULL && holder->groupLeader == leader)
+ continue;
+
+ /* Must the holder later wait for me? */
+ if ((lockMethodTable->conflictTab[holder->upgradeIntent] &
+ myHeldLocks) == 0)
+ continue;
+
+ /* Must I wait for the holder now? */
+ if ((lockMethodTable->conflictTab[lockmode] &
+ holder->holdMask) == 0)
+ continue;
+
+ return PROC_WAIT_STATUS_PREEMPTED;
+ }
+ }
+
/*
* At this point we know that we'd really need to sleep. If we've been
* commanded not to do that, bail out.
@@ -1308,8 +1385,10 @@ JoinWaitQueue(LOCALLOCK *locallock, LockMethod lockMethodTable, bool dontWait)
*
* Result is one of the following:
*
- * PROC_WAIT_STATUS_OK - lock was granted
- * PROC_WAIT_STATUS_ERROR - a deadlock was detected
+ * PROC_WAIT_STATUS_OK - lock granted
+ * PROC_WAIT_STATUS_ERROR - deadlock detected
+ * PROC_WAIT_STATUS_PREEMPTED - canceled by another backend's DeadLockCheck
+ * to break a protected-upgrade cycle
*/
ProcWaitStatus
ProcSleep(LOCALLOCK *locallock)
@@ -1562,16 +1641,19 @@ ProcSleep(LOCALLOCK *locallock)
/*
* If awoken after the deadlock check interrupt has run, increment the
- * lock statistics counters and if log_lock_waits is on, then report
- * about the wait.
+ * lock statistics counters and, if log_lock_waits is on, report about
+ * the wait. Also report a wait that another backend preempted before
+ * our own deadlock timeout fired.
*/
- if (deadlock_state != DS_NOT_YET_CHECKED)
+ if (deadlock_state != DS_NOT_YET_CHECKED ||
+ (myWaitStatus == PROC_WAIT_STATUS_PREEMPTED && log_lock_waits))
{
long secs;
int usecs;
long msecs;
- INJECTION_POINT("deadlock-timeout-fired", NULL);
+ if (deadlock_state != DS_NOT_YET_CHECKED)
+ INJECTION_POINT("deadlock-timeout-fired", NULL);
TimestampDifference(get_timeout_start_time(DEADLOCK_TIMEOUT),
GetCurrentTimestamp(),
&secs, &usecs);
@@ -1611,6 +1693,13 @@ ProcSleep(LOCALLOCK *locallock)
(errdetail_log_plural("Process holding the lock: %s. Wait queue: %s.",
"Processes holding the lock: %s. Wait queue: %s.",
lockHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data))));
+ else if (deadlock_state == DS_PREEMPT_DEADLOCK)
+ ereport(LOG,
+ (errmsg("process %d avoided deadlock for %s on %s by canceling a blocking lock wait after %ld.%03d ms",
+ MyProcPid, modename, buf.data, msecs, usecs),
+ (errdetail_log_plural("Process holding the lock: %s. Wait queue: %s.",
+ "Processes holding the lock: %s. Wait queue: %s.",
+ lockHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data))));
else if (deadlock_state == DS_HARD_DEADLOCK)
{
/*
@@ -1659,16 +1748,13 @@ ProcSleep(LOCALLOCK *locallock)
MyProcPid, modename, buf.data, msecs, usecs)));
else
{
- Assert(myWaitStatus == PROC_WAIT_STATUS_ERROR);
+ Assert(myWaitStatus == PROC_WAIT_STATUS_ERROR ||
+ myWaitStatus == PROC_WAIT_STATUS_PREEMPTED);
/*
- * Currently, the deadlock checker always kicks its own
- * process, which means that we'll only see
- * PROC_WAIT_STATUS_ERROR when deadlock_state ==
- * DS_HARD_DEADLOCK, and there's no need to print
- * redundant messages. But for completeness and
- * future-proofing, print a message if it looks like
- * someone else kicked us off the lock.
+ * Self-detected hard deadlocks log via DeadLockReport;
+ * skip the redundant LOG. Otherwise (e.g. preempted),
+ * emit it here.
*/
if (deadlock_state != DS_HARD_DEADLOCK)
ereport(LOG,
@@ -1870,14 +1956,16 @@ CheckDeadLock(void)
* Get this process out of wait state. (Note: we could do this more
* efficiently by relying on lockAwaited, but use this coding to
* preserve the flexibility to kill some other transaction than the
- * one detecting the deadlock.)
+ * one detecting the deadlock. DS_PREEMPT_DEADLOCK exercises that
+ * already.)
*
* RemoveFromWaitQueue sets MyProc->waitStatus to
* PROC_WAIT_STATUS_ERROR, so ProcSleep will report an error after we
* return.
*/
Assert(MyProc->waitLock != NULL);
- RemoveFromWaitQueue(MyProc, LockTagHashCode(&(MyProc->waitLock->tag)));
+ RemoveFromWaitQueue(MyProc, LockTagHashCode(&(MyProc->waitLock->tag)),
+ PROC_WAIT_STATUS_ERROR);
/*
* We're done here. Transaction abort caused by the error that
diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h
index 9453a3e4932..f0a71184757 100644
--- a/src/include/catalog/namespace.h
+++ b/src/include/catalog/namespace.h
@@ -99,6 +99,13 @@ typedef void (*RangeVarGetRelidCallback) (const RangeVar *relation, Oid relId,
RangeVarGetRelidExtended(relation, lockmode, \
(missing_ok) ? RVR_MISSING_OK : 0, NULL, NULL)
+extern Oid RangeVarGetRelidWithUpgradeIntent(const RangeVar *relation,
+ LOCKMODE lockmode,
+ LOCKMODE upgradeMode,
+ uint32 flags,
+ RangeVarGetRelidCallback callback,
+ void *callback_arg);
+
extern Oid RangeVarGetRelidExtended(const RangeVar *relation,
LOCKMODE lockmode, uint32 flags,
RangeVarGetRelidCallback callback,
diff --git a/src/include/storage/lmgr.h b/src/include/storage/lmgr.h
index 2a985ce5e15..57667432e02 100644
--- a/src/include/storage/lmgr.h
+++ b/src/include/storage/lmgr.h
@@ -42,6 +42,9 @@ extern void LockRelationId(LockRelId *relid, LOCKMODE lockmode);
extern bool ConditionalLockRelationOid(Oid relid, LOCKMODE lockmode);
extern void UnlockRelationId(LockRelId *relid, LOCKMODE lockmode);
extern void UnlockRelationOid(Oid relid, LOCKMODE lockmode);
+extern void LockRelationOidWithUpgradeIntent(Oid relid,
+ LOCKMODE lockmode,
+ LOCKMODE upgradeMode);
extern void LockRelation(Relation relation, LOCKMODE lockmode);
extern bool ConditionalLockRelation(Relation relation, LOCKMODE lockmode);
diff --git a/src/include/storage/lock.h b/src/include/storage/lock.h
index ee3cb1dc203..cff76a7aed8 100644
--- a/src/include/storage/lock.h
+++ b/src/include/storage/lock.h
@@ -131,6 +131,7 @@ typedef const LockMethodData *LockMethod;
* nRequested -- total requested locks of all types.
* granted -- count of each lock type currently granted on the lock.
* nGranted -- total granted locks of all types.
+ * nUpgradeIntent -- # of proclocks here with upgradeIntent set.
*
* Note: these counts count 1 for each backend. Internally to a backend,
* there may be multiple grabs on a particular lock, but this is not reflected
@@ -150,6 +151,7 @@ typedef struct LOCK
int nRequested; /* total of requested[] array */
int granted[MAX_LOCKMODES]; /* counts of granted locks */
int nGranted; /* total of granted[] array */
+ int nUpgradeIntent; /* see LOCK comment above */
} LOCK;
#define LOCK_LOCKMETHOD(lock) ((LOCKMETHODID) (lock).tag.locktag_lockmethodid)
@@ -206,6 +208,7 @@ typedef struct PROCLOCK
PGPROC *groupLeader; /* proc's lock group leader, or proc itself */
LOCKMASK holdMask; /* bitmask for lock types currently held */
LOCKMASK releaseMask; /* bitmask for lock types to be released */
+ LOCKMODE upgradeIntent; /* announced future upgrade target, or NoLock */
dlist_node lockLink; /* list link in LOCK's list of proclocks */
dlist_node procLink; /* list link in PGPROC's list of proclocks */
} PROCLOCK;
@@ -342,10 +345,22 @@ typedef enum
DS_NO_DEADLOCK, /* no deadlock detected */
DS_SOFT_DEADLOCK, /* deadlock avoided by queue rearrangement */
DS_HARD_DEADLOCK, /* deadlock, no way out but ERROR */
+ DS_PREEMPT_DEADLOCK, /* resolved by canceling a blocking waiter */
DS_BLOCKED_BY_AUTOVACUUM, /* no deadlock; queue blocked by autovacuum
* worker */
} DeadLockState;
+/*
+ * Lock-wait status set by RemoveFromWaitQueue and read by the waiter on wakeup
+ */
+typedef enum
+{
+ PROC_WAIT_STATUS_OK,
+ PROC_WAIT_STATUS_WAITING,
+ PROC_WAIT_STATUS_ERROR,
+ PROC_WAIT_STATUS_PREEMPTED,
+} ProcWaitStatus;
+
/*
* The lockmgr's shared hash tables are partitioned to reduce contention.
* To determine which partition a given locktag belongs to, compute the tag's
@@ -386,6 +401,7 @@ extern LockAcquireResult LockAcquire(const LOCKTAG *locktag,
bool dontWait);
extern LockAcquireResult LockAcquireExtended(const LOCKTAG *locktag,
LOCKMODE lockmode,
+ LOCKMODE upgradeIntent,
bool sessionLock,
bool dontWait,
bool reportMemoryError,
@@ -418,7 +434,8 @@ extern void GrantAwaitedLock(void);
extern LOCALLOCK *GetAwaitedLock(void);
extern void ResetAwaitedLock(void);
-extern void RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode);
+extern void RemoveFromWaitQueue(PGPROC *proc, uint32 hashcode,
+ ProcWaitStatus newStatus);
extern LockData *GetLockStatusData(void);
extern BlockedProcsData *GetBlockerStatusData(int blocked_pid);
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 3e1d1fad5f9..9f89e587f3a 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -146,13 +146,6 @@ extern PGDLLIMPORT int FastPathLockGroupsPerBackend;
#define DELAY_CHKPT_COMPLETE (1<<1)
#define DELAY_CHKPT_IN_COMMIT (DELAY_CHKPT_START | 1<<2)
-typedef enum
-{
- PROC_WAIT_STATUS_OK,
- PROC_WAIT_STATUS_WAITING,
- PROC_WAIT_STATUS_ERROR,
-} ProcWaitStatus;
-
/*
* Each backend has a PGPROC struct in shared memory. There is also a list of
* currently-unused PGPROC structs that will be reallocated to new backends.
diff --git a/src/test/modules/injection_points/expected/repack.out b/src/test/modules/injection_points/expected/repack.out
index b575e9052ee..9dffc603d9f 100644
--- a/src/test/modules/injection_points/expected/repack.out
+++ b/src/test/modules/injection_points/expected/repack.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 4 sessions
starting permutation: wait_before_lock change_existing change_new change_subxact1 change_subxact2 check2 wakeup_before_lock check1
injection_points_attach
@@ -111,3 +111,182 @@ injection_points_detach
(1 row)
+
+starting permutation: wait_before_lock begin_txn lock_table_ae s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_txn:
+ BEGIN;
+
+step lock_table_ae:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step lock_table_ae: <... completed>
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read lock_table_ae s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table_ae:
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step wait_before_lock: <... completed>
+step lock_table_ae: <... completed>
+ERROR: deadlock detected
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read lock_table_sue s3_wakeup end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step lock_table_sue:
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step s3_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step lock_table_sue: <... completed>
+ERROR: deadlock detected
+step wait_before_lock: <... completed>
+step end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
+
+starting permutation: s2_timeout_fast wait_before_lock begin_and_read s3_begin_txn s3_lock_table_y_sue s3_lock_table_sue lock_table_y_sue s4_wakeup end_txn s3_end_txn
+injection_points_attach
+-----------------------
+
+(1 row)
+
+step s2_timeout_fast:
+ SET deadlock_timeout = '10ms';
+
+step wait_before_lock:
+ REPACK (CONCURRENTLY) repack_test USING INDEX repack_test_pkey;
+ <waiting ...>
+step begin_and_read:
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_begin_txn:
+ BEGIN;
+
+step s3_lock_table_y_sue:
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+
+step s3_lock_table_sue:
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step lock_table_y_sue:
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+ <waiting ...>
+step s4_wakeup:
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+
+injection_points_wakeup
+-----------------------
+
+(1 row)
+
+step lock_table_y_sue: <... completed>
+step s3_lock_table_sue: <... completed>
+ERROR: deadlock detected
+step end_txn:
+ COMMIT;
+
+step wait_before_lock: <... completed>
+step s3_end_txn:
+ COMMIT;
+
+injection_points_detach
+-----------------------
+
+(1 row)
+
diff --git a/src/test/modules/injection_points/specs/repack.spec b/src/test/modules/injection_points/specs/repack.spec
index d727a9b056b..36d32964f94 100644
--- a/src/test/modules/injection_points/specs/repack.spec
+++ b/src/test/modules/injection_points/specs/repack.spec
@@ -5,6 +5,7 @@ setup
CREATE TABLE repack_test(i int PRIMARY KEY, j int);
INSERT INTO repack_test(i, j) VALUES (1, 1), (2, 2), (3, 3), (4, 4);
+ CREATE TABLE repack_test_y(i int);
CREATE TABLE relfilenodes(node oid);
@@ -15,6 +16,7 @@ setup
teardown
{
DROP TABLE repack_test;
+ DROP TABLE repack_test_y;
DROP EXTENSION injection_points;
DROP TABLE relfilenodes;
@@ -61,6 +63,10 @@ teardown
}
session s2
+step s2_timeout_fast
+{
+ SET deadlock_timeout = '10ms';
+}
# Change the existing data. UPDATE changes both key and non-key columns. Also
# update one row twice to test whether tuple version generated by this session
# can be found.
@@ -128,6 +134,60 @@ step wakeup_before_lock
{
SELECT injection_points_wakeup('repack-concurrently-before-lock');
}
+# Steps used in lock contention tests.
+step begin_txn
+{
+ BEGIN;
+}
+step begin_and_read
+{
+ BEGIN;
+ SELECT 1 FROM repack_test LIMIT 1;
+}
+step lock_table_ae
+{
+ LOCK TABLE repack_test IN ACCESS EXCLUSIVE MODE;
+}
+step lock_table_sue
+{
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step lock_table_y_sue
+{
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step end_txn
+{
+ COMMIT;
+}
+
+session s3
+step s3_begin_txn
+{
+ BEGIN;
+}
+step s3_lock_table_y_sue
+{
+ LOCK TABLE repack_test_y IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step s3_lock_table_sue
+{
+ LOCK TABLE repack_test IN SHARE UPDATE EXCLUSIVE MODE;
+}
+step s3_wakeup
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
+step s3_end_txn
+{
+ COMMIT;
+}
+
+session s4
+step s4_wakeup
+{
+ SELECT injection_points_wakeup('repack-concurrently-before-lock');
+}
# Test if data changes introduced while one session is performing REPACK
# CONCURRENTLY find their way into the table.
@@ -140,3 +200,51 @@ permutation
check2
wakeup_before_lock
check1
+
+# A waiter that does not already hold a conflicting lock on the table is
+# waits until REPACK finishes and then acquires its lock (soft-deadlock).
+permutation
+ wait_before_lock
+ begin_txn
+ lock_table_ae
+ s3_wakeup
+ end_txn
+
+# A waiter that already holds AccessShareLock then waits for AccessExclusiveLock
+# behind REPACK's ShareUpdateExclusiveLock.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ lock_table_ae(*)
+ s3_wakeup
+ end_txn
+
+# Same shape as above, but the waiter requests ShareUpdateExclusiveLock.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ lock_table_sue(*)
+ s3_wakeup
+ end_txn
+
+# Three-backend future deadlock:
+#
+# - s1 holds ShareUpdateExclusiveLock on repack_test and has declared a future
+# AccessExclusiveLock on it.
+# - s2 holds AccessShareLock on repack_test.
+# - s3 holds ShareUpdateExclusiveLock on repack_test_y, then waits for
+# ShareUpdateExclusiveLock on repack_test behind s1.
+# - s2 waits for ShareUpdateExclusiveLock on repack_test_y behind s3.
+permutation
+ s2_timeout_fast
+ wait_before_lock
+ begin_and_read
+ s3_begin_txn
+ s3_lock_table_y_sue
+ s3_lock_table_sue
+ lock_table_y_sue(*)
+ s4_wakeup
+ end_txn
+ s3_end_txn
--
2.43.0
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-19 11:00 ` Alexander Lakhin <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Alexander Lakhin @ 2026-04-19 11:00 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Andres Freund <[email protected]>; Mihail Nikalayeu <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello Antonin and Alvaro,
I've noticed a wrong reference coined in commit 28d534e2a:
A comment in repack_worker.c says:
/*
* Override the default bgworker_die() with die() so we can use
* CHECK_FOR_INTERRUPTS().
*/
pqsignal(SIGTERM, die);
BackgroundWorkerUnblockSignals();
but bgworker_die() doesn't exist since d62dca3b2 (2026-02-18).
Maybe this is not needed anymore?
Best regards,
Alexander
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-17 14:44 ` Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Justin Pryzby @ 2026-04-17 14:44 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Antonin Houska <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
I had trouble testing this at first:
postgres=# REPACK (CONCURRENTLY) users;
ERROR: 42501: permission denied to use replication slots
DETAIL: Only roles with the REPLICATION attribute may use replication slots.
CONTEXT: REPACK decoding worker
LOCATION: CheckSlotPermissions, slot.c:1697
That's surprising, since it was run as a superuser.
It turns out that repack runs as the owner of the table, and the table
*owner* needs to have REPLICATION -- regardless of who runs the command.
I imagine people have been testing with one user, that both owns the
table and invokes REPACK. Maybe this just needs to be clarified in the
documentation/error message?
--
Justin
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
@ 2026-04-20 07:44 ` Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-20 07:44 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
Justin Pryzby <[email protected]> wrote:
> I had trouble testing this at first:
>
> postgres=# REPACK (CONCURRENTLY) users;
> ERROR: 42501: permission denied to use replication slots
> DETAIL: Only roles with the REPLICATION attribute may use replication slots.
> CONTEXT: REPACK decoding worker
> LOCATION: CheckSlotPermissions, slot.c:1697
>
> That's surprising, since it was run as a superuser.
>
> It turns out that repack runs as the owner of the table, and the table
> *owner* needs to have REPLICATION -- regardless of who runs the command.
>
> I imagine people have been testing with one user, that both owns the
> table and invokes REPACK. Maybe this just needs to be clarified in the
> documentation/error message?
It was discussed earlier [1] and the concerns about possibly excessive
resource consumptions were addressed by [2]. So I think it the fix was just
forgotten. Attached here.
[1] https://www.postgresql.org/message-id/[email protected]
[2] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=e76d8c749c3152657711ed733f0aea61c0e36...
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-20 07:47 ` Antonin Houska <[email protected]>
2026-04-20 07:54 ` Re: Adding REPACK [concurrently] Chao Li <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-20 07:47 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
Antonin Houska <[email protected]> wrote:
> It was discussed earlier [1] and the concerns about possibly excessive
> resource consumptions were addressed by [2]. So I think it the fix was just
> forgotten. Attached here.
Sorry, I attached wrong patch. This is what I meant.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-20 07:54 ` Chao Li <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Chao Li @ 2026-04-20 07:54 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
> On Apr 20, 2026, at 15:47, Antonin Houska <[email protected]> wrote:
>
> Antonin Houska <[email protected]> wrote:
>
>> It was discussed earlier [1] and the concerns about possibly excessive
>> resource consumptions were addressed by [2]. So I think it the fix was just
>> forgotten. Attached here.
>
> Sorry, I attached wrong patch. This is what I meant.
>
> --
> Antonin Houska
> Web: https://www.cybertec-postgresql.com
>
> <0001-Do-not-check-the-REPLICATION-attribute-when-running-.patch>
In my test, I found the exact same issue and I was about to post a fix that is the same as v1. So, v1 looks good to me.
BTW, Antonin, can you please take a look at [1] that is the other issue I found with repack.
[1] https://www.postgresql.org/message-id/10DD5E13-B45D-44F1-BE08-C63E00ABCAC0%40gmail.com
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-20 11:45 ` Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-20 11:45 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-20, Antonin Houska wrote:
> Antonin Houska <[email protected]> wrote:
>
> > It was discussed earlier [1] and the concerns about possibly excessive
> > resource consumptions were addressed by [2]. So I think it the fix was just
> > forgotten. Attached here.
>
> Sorry, I attached wrong patch. This is what I meant.
Yeah, I had also written the same patch a couple of days ago.
BTW I ran into a small problem after adding some tests in cluster.sql
that would exercise this -- that test would die more or less randomly
but frequently in CI (which it never did in my laptop) because of the
size of the snapshot,
ALTER TABLE ptnowner1 REPLICA IDENTITY USING INDEX ptnowner1_i_key;
REPACK (CONCURRENTLY) ptnowner1;
+ERROR: initial slot snapshot too large
+CONTEXT: REPACK decoding worker
RESET SESSION AUTHORIZATION;
I think the solution for this is to move cluster to a separate parallel
test. The one where it is now is a bit too crowded. Maybe the one for
compression is okay? I'll test and push if I see it passing CI.
Another obvious thing after adding tests is that the LOGIN privilege is
required, which is also quite bogus IMO. 0002 here should solve that.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"If you want to have good ideas, you must have many ideas. Most of them
will be wrong, and what you have to learn is which ones to throw away."
(Linus Pauling)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-20 13:46 ` Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Antonin Houska @ 2026-04-20 13:46 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> On 2026-Apr-20, Antonin Houska wrote:
>
> > Antonin Houska <[email protected]> wrote:
> >
> > > It was discussed earlier [1] and the concerns about possibly excessive
> > > resource consumptions were addressed by [2]. So I think it the fix was just
> > > forgotten. Attached here.
> >
> > Sorry, I attached wrong patch. This is what I meant.
>
> Yeah, I had also written the same patch a couple of days ago.
>
> BTW I ran into a small problem after adding some tests in cluster.sql
> that would exercise this -- that test would die more or less randomly
> but frequently in CI (which it never did in my laptop) because of the
> size of the snapshot,
>
> ALTER TABLE ptnowner1 REPLICA IDENTITY USING INDEX ptnowner1_i_key;
> REPACK (CONCURRENTLY) ptnowner1;
> +ERROR: initial slot snapshot too large
> +CONTEXT: REPACK decoding worker
> RESET SESSION AUTHORIZATION;
>
> I think the solution for this is to move cluster to a separate parallel
> test. The one where it is now is a bit too crowded. Maybe the one for
> compression is okay? I'll test and push if I see it passing CI.
That shouldn't break anything, but I have no idea why this problem was not
triggered (as far as I remember) by the stress tests we ran during
development.
I thought that it might be due to less frequent calls of
SnapBuildPurgeOlderTxn(), which might in turn be due to less frequent
checkpoints (because xl_running_xacts is typically written during checkpoint),
and that checkpoints may be deliberately less frequent to make regression
tests run faster. However I don't immediately see related configuration in the
regression test setup.
> Another obvious thing after adding tests is that the LOGIN privilege is
> required, which is also quite bogus IMO. 0002 here should solve that.
ok
> >From b3d4158356f4914d2b0cba86eef6994c0ee50ab9 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <[email protected]>
> Date: Mon, 20 Apr 2026 11:38:48 +0200
> Subject: [PATCH 1/2] REPACK: do not require the user to have REPLICATION
> Because there are now successful concurrent repack runs in the
> regression tests, we're forced to run test_plan_advice under
> wal_level=replica.
Is this an attempt to disable REPACK (CONCURRENTLY)? That would require
wal_level=minimal (due to commit 67c20979ce). In which way does REPACK seem to
break test_plan_advice?
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-20 13:54 ` Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-20 13:54 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-20, Antonin Houska wrote:
> Alvaro Herrera <[email protected]> wrote:
> > BTW I ran into a small problem after adding some tests in cluster.sql
> > that would exercise this -- that test would die more or less randomly
> > but frequently in CI (which it never did in my laptop) because of the
> > size of the snapshot,
> >
> > ALTER TABLE ptnowner1 REPLICA IDENTITY USING INDEX ptnowner1_i_key;
> > REPACK (CONCURRENTLY) ptnowner1;
> > +ERROR: initial slot snapshot too large
> > +CONTEXT: REPACK decoding worker
> > RESET SESSION AUTHORIZATION;
> >
> > I think the solution for this is to move cluster to a separate parallel
> > test. The one where it is now is a bit too crowded. Maybe the one for
> > compression is okay? I'll test and push if I see it passing CI.
>
> That shouldn't break anything, but I have no idea why this problem was not
> triggered (as far as I remember) by the stress tests we ran during
> development.
I took a guess that it's because some tests use minimally configured
servers -- that is, max_connections=20 or so -- and then run 20
processes. If we then try to construct a snapshot that's limited to
having only that many XIDs, we might not have enough room in the xip
array. I didn't try to trace it super carefully though.
> > >From b3d4158356f4914d2b0cba86eef6994c0ee50ab9 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <[email protected]>
> > Date: Mon, 20 Apr 2026 11:38:48 +0200
> > Subject: [PATCH 1/2] REPACK: do not require the user to have REPLICATION
>
> > Because there are now successful concurrent repack runs in the
> > regression tests, we're forced to run test_plan_advice under
> > wal_level=replica.
>
> Is this an attempt to disable REPACK (CONCURRENTLY)? That would require
> wal_level=minimal (due to commit 67c20979ce). In which way does REPACK seem to
> break test_plan_advice?
No, quite the contrary. That test normally runs with wal_level=minimal,
which causes REPACK to complain that it cannot start logical
decoding.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-20 14:42 ` Tom Lane <[email protected]>
2026-04-20 15:11 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Tom Lane @ 2026-04-20 14:42 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> writes:
> On 2026-Apr-20, Antonin Houska wrote:
>> Is this an attempt to disable REPACK (CONCURRENTLY)? That would require
>> wal_level=minimal (due to commit 67c20979ce). In which way does REPACK seem to
>> break test_plan_advice?
> No, quite the contrary. That test normally runs with wal_level=minimal,
> which causes REPACK to complain that it cannot start logical
> decoding.
So what you're saying is that the core regression tests will now fail
with wal_level=minimal? I don't see how that can possibly be
considered acceptable from a global standpoint; we might as well
remove wal_level=minimal, because it will never again get tested.
I think you need to move these tests out into some other test suite
(or make a new one).
regards, tom lane
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
@ 2026-04-20 15:11 ` Alvaro Herrera <[email protected]>
2026-04-20 16:01 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-20 16:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-20 15:11 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-20, Tom Lane wrote:
> So what you're saying is that the core regression tests will now fail
> with wal_level=minimal? I don't see how that can possibly be
> considered acceptable from a global standpoint; we might as well
> remove wal_level=minimal, because it will never again get tested.
Hmm, you're right, this was brain fade on my part.
It surprising though that no buildfarm animal so far has complained.
But I remember there's at least one that runs with that, per
be142fa008ad.
> I think you need to move these tests out into some other test suite
> (or make a new one).
I'll see what I can find.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-20 15:11 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-20 16:01 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Tom Lane @ 2026-04-20 16:01 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> writes:
> It surprising though that no buildfarm animal so far has complained.
> But I remember there's at least one that runs with that, per
> be142fa008ad.
Yeah, thorntail does, but it only runs once a day.
regards, tom lane
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-20 15:11 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-20 16:30 ` Alvaro Herrera <[email protected]>
2026-04-22 09:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-20 16:30 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-20, Alvaro Herrera wrote:
> On 2026-Apr-20, Tom Lane wrote:
> > I think you need to move these tests out into some other test suite
> > (or make a new one).
>
> I'll see what I can find.
I think the simplest would be to add them to
src/test/modules/injection_points. We already have some repack tests
there (because they needed injection points), and it has an SQL suite,
so we would not be adding any extra overhead. However, the new tests
are not related to injection points, so they would be out of place.
Another possibility could be src/test/subscription, but it doesn't have
sql tests; doesn't seem good to have them just for this.
There's also contrib/test_decoding. It's somewhat vaguely adjacent, and
the tests aren't _really_ about the decoding part, but of all these
options, it seems the least bad one.
I don't like the idea of adding another suite. Too much scaffolding for
so little, I think.
I don't have any other ideas ATM.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"We're here to devour each other alive" (Hobbes)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-20 15:11 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 16:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-22 09:30 ` Alvaro Herrera <[email protected]>
2026-04-23 11:01 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-22 09:30 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-20, Alvaro Herrera wrote:
> There's also contrib/test_decoding. It's somewhat vaguely adjacent, and
> the tests aren't _really_ about the decoding part, but of all these
> options, it seems the least bad one.
Here's a patch that does this.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-15 14:59 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 18:28 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-16 11:18 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-17 14:44 ` Re: Adding REPACK [concurrently] Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 07:47 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 11:45 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-20 13:54 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Re: Adding REPACK [concurrently] Tom Lane <[email protected]>
2026-04-20 15:11 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-20 16:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-22 09:30 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-23 11:01 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 156+ messages in thread
From: Alvaro Herrera @ 2026-04-23 11:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Antonin Houska <[email protected]>; Justin Pryzby <[email protected]>; Mihail Nikalayeu <[email protected]>; Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; [email protected]; Robert Treat <[email protected]>
On 2026-Apr-22, Alvaro Herrera wrote:
> On 2026-Apr-20, Alvaro Herrera wrote:
>
> > There's also contrib/test_decoding. It's somewhat vaguely adjacent, and
> > the tests aren't _really_ about the decoding part, but of all these
> > options, it seems the least bad one.
>
> Here's a patch that does this.
Pushed, thanks. Thorntail should turn green in its next run.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Nunca se desea ardientemente lo que solo se desea por razón" (F. Alexandre)
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-09 08:43 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-09 14:20 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-12 13:31 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-14 13:37 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-14 13:58 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-15 14:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-15 15:54 ` Mihail Nikalayeu <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Mihail Nikalayeu @ 2026-04-15 15:54 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Andres Freund <[email protected]>; Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hello!
On Wed, Apr 15, 2026 at 4:50 PM Antonin Houska <[email protected]> wrote:
> The approach proposed by Mihail [1] seems the least problematic to me, and
> something like that occurred to me when I thought about the problem the first
> time. However, when we wake up the other processes in order to run the
> deadlock detection, they should do that immediately. I've got no good idea
> about implementation at the moment, since latch can be set for unrelated
> reasons. (Besides that, I have some more questions about this patch, which I
> can post separately.)
It is already possible to "deadlock" ANOTHER backend while running the
deadlock check [0].
Supported in current infra, just not used at the moment (my POC used that also).
[0]: https://github.com/postgres/postgres/blob/master/src/backend/storage/lmgr/proc.c#L1870-L1873
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 10:14 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-07 20:24 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 08:35 ` Re: Adding REPACK [concurrently] Amit Kapila <[email protected]>
2026-04-08 17:22 ` Re: Adding REPACK [concurrently] Andres Freund <[email protected]>
2026-04-08 23:36 ` Re: Adding REPACK [concurrently] Mihail Nikalayeu <[email protected]>
@ 2026-04-09 14:13 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Andres Freund @ 2026-04-09 14:13 UTC (permalink / raw)
To: Mihail Nikalayeu <[email protected]>; +Cc: Amit Kapila <[email protected]>; Alvaro Herrera <[email protected]>; Antonin Houska <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi,
On 2026-04-09 01:36:00 +0200, Mihail Nikalayeu wrote:
> Hello, Andres!
>
> On Wed, Apr 8, 2026 at 7:22 PM Andres Freund <[email protected]> wrote:
> > I don't think this is a viable path. You need to prevent any further lock
> > acquisitions on the relation to be able to swap it, not just conflicting DDL.
>
> AFAIU, Amit's main idea is that we currently upgrade the lock instead
> of **releasing and re-acquiring** it because we fear DDL between those
> actions.
I got that, I just don't think it's going to work in any sort of way
reasonably well.
The fix here should be to make the system understand how to make lock upgrades
as safe as possible, not to hack around the corners. For that you need to
teach the deadlock detector about all of this properly.
> > I don't think CheckTableNotInUse() would work anyway - don't we already hold
> > locks by the point we call it?
>
> Yes, the DDL session already holds locks by the time
> CheckTableNotInUse is called - but is that really the problem? They
> will be released on error.
There's no guarantee they get there if it's done after the lock
acquisition. It can be part of a more complicated lock cycle.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-01 21:52 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-03 17:24 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 09:48 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-04 23:53 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 11:50 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
2026-04-05 18:56 ` Re: Adding REPACK [concurrently] Alvaro Herrera <[email protected]>
@ 2026-04-06 09:03 ` Antonin Houska <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Antonin Houska @ 2026-04-06 09:03 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Alvaro Herrera <[email protected]> wrote:
> So I've been trying to understand the "Introduce an option to make
> logical replication database specific." patch and I have to confess I
> just cannot.
>
> As far as I can read, the point is that if we reach
> SnapBuildProcessRunningXacts() when db_specific is true (which means
> standby_decode is called in an output plugin that has set
> need_shared_catalogs to false), _and_ we've not reached consistent state
> yet, then we'll call LogStandbySnapshot with our DB oid to emit a new
> xl_running_xacts message.
Right.
> So the WAL-decoding process emits WAL. I don't know if in normal
> conditions logical decoding processes emit WAL. If this is exceptional,
> I think we should add a comment.
Emitting WAL in logical decoding is not exceptional: SnapBuildWaitSnapshot()
already calls LogStandbySnapshot(), in order to get to the next phase.
> Now, this additional WAL message will be processed by all other
> processes decoding WAL. Perhaps it will ignored by most of them.
Right. That's one thing that I realized late yesterday, after having posted
the latest version of the patch. In SnapBuildProcessRunningXacts(), we need
if (OidIsValid(running->dbid))
return;
rather than
if (db_specific)
return;
because other backends can also generate their database-specific records.
> But
> most importantly, it will also reach back to ourselves, at which point
> we can hopefully use it to see that we might have reached consistent
> state within our database. Then we know our snapshot is ready to be
> used.
>
> Is this correct?
Yes.
> I think the reason it's safe to skip a lot of the processing caused by
> this additional process, is that xl_running_xacts messages are also
> emitted in other places in a non-database specific manner. So all the
> other placecs that are emitting that message continue to exist and
> cause logical-decoders operate in the same way as before.
Yes. I'm still thinking though if, after having used the database-specific
record to reach CONSITENT, we sould enforce one cluster-wide record, so that
the cleanup (in "our backend") takes place sooner rather than later. Not sure
about that.
> I think we should sprinkle lots of comments in several places about
> this. For example, I propose that standby_redo() should have something
> like
>
> * If 'dbid' is valid, only gather transactions running in that database.
> + * Such records should not be the only ones emitted, because this has
> + * potentially dangerous side-effects which makes some places ignore them:
> + *
> + * 1. SnapBuildProcessRunningXacts will skip computing the xmin and restart
> + * point from its input record if the record's xmin is older that the
> + * snapbuilder's current xmin; this should normally be fine because that
> + * information will be updated from other xl_running_xacts records.
> + * 2. standby_redo will likewise skip processing such a record
> *
> (are there other things that should be mentioned?)
I added something like that, but - due to the reference to
SnapBuildProcessRunningXacts() - less verbose about snapbuild.c.
> Also, LogStandbySnapshot() should have a comment explaining that passing
> a valid dboid is a weird corner case which is to be used with care, and
> that functions X Y and Z are going to ignore snapshots carrying a valid
> dbid.
One more check added in standby_decode() (and mentioned in that function in
the comment).
> Why do we call SnapBuildFindSnapshot() to do this, instead of doing it
> directly in SnapBuildProcessRunningXacts? Seems like it would be more
> straightforward.
Yes, fixed.
One more problem I found when testing w/o background worker
(contrib/test_decoding) was that accessSharedCatalogsInDecoding was not set
back to true. Both AllocateSnapshotBuilder() FreeSnapshotBuilder() do that
now.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 156+ messages in thread
* Re: Adding REPACK [concurrently]
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Re: Adding REPACK [concurrently] Antonin Houska <[email protected]>
@ 2026-04-02 00:19 ` Srinath Reddy Sadipiralla <[email protected]>
1 sibling, 0 replies; 156+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-04-02 00:19 UTC (permalink / raw)
To: Antonin Houska <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Amit Kapila <[email protected]>; Mihail Nikalayeu <[email protected]>; Matthias van de Meent <[email protected]>; Pg Hackers <[email protected]>; Robert Treat <[email protected]>
Hi Antonin,
On Wed, Apr 1, 2026 at 10:36 PM Antonin Houska <[email protected]> wrote:
> Srinath Reddy Sadipiralla <[email protected]> wrote:
>
> > i was fuzz testing v48 , and found a crash when REPACK (concurrently)
> itself errors out,
> > 1) while running
> >
> > create table test(id int);
> > REPACK (concurrently) test;
> >
> > TBH i didn't knew this, sometimes it's better to not know "rules" ;)
> > [NOTE: maybe we should add that we can't run
> > REPACK (concurrently) on table without identity index or primary key
> into repack.sgml]
> >
> > ERROR: cannot process relation "test"
> > 2026-04-01 19:06:31.211 IST [2495575] HINT: Relation "test" has no
> identity index.
> > 2026-04-01 19:06:31.211 IST [2495575] STATEMENT: repack (concurrently)
> test;
> > TRAP: failed Assert("proc->statusFlags ==
> ProcGlobal->statusFlags[proc->pgxactoff]"), File: "procarray.c", Line: 719,
> PID: 2495575
> > Here's the diff to solve this crash.
>
> Thanks. Attached here is v48-0006 fixed.
>
On Wed, Apr 1, 2026 at 8:25 PM Srinath Reddy Sadipiralla <
[email protected]> wrote:
> Here's the diff to solve this crash.
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
> index 29ba49744eb..d44092a407a 100644
> --- a/src/backend/commands/repack.c
> +++ b/src/backend/commands/repack.c
> @@ -284,7 +284,23 @@ ExecRepack(ParseState *pstate, RepackStmt *stmt, bool
> isTopLevel)
> * that others can conflict with.
> */
> if ((params.options & CLUOPT_CONCURRENT) != 0)
> + {
> + /*
> + * Do not let other backends wait for our completion during their
> + * setup of logical replication. Unlike logical replication publisher,
> + * we will have XID assigned, so the other backends - whether
> + * walsenders involved in logical replication or regular backends
> + * executing also REPACK (CONCURRENTLY) - would have to wait for our
> + * completion before they can build their initial snapshot. It is o.k.
+ * for any decoding backend to ignore us because we do not change
> + * tuple descriptor of any table, and the data changes we write should
> + * not be decoded by other backends.
> + */
> + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
> MyProc->statusFlags |= PROC_IN_CONCURRENT_REPACK;
> + ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags;
> + LWLockRelease(ProcArrayLock);
> + }
>
> /*
> * If a single relation is specified, process it and we're done ... unless
> @@ -988,22 +1004,6 @@ rebuild_relation(Relation OldHeap, Relation index,
> bool verbose,
>
> if (concurrent)
> {
> - /*
> - * Do not let other backends wait for our completion during their
> - * setup of logical replication. Unlike logical replication publisher,
> - * we will have XID assigned, so the other backends - whether
> - * walsenders involved in logical replication or regular backends
> - * executing also REPACK (CONCURRENTLY) - would have to wait for our
> - * completion before they can build their initial snapshot. It is o.k.
> - * for any decoding backend to ignore us because we do not change
> - * tuple descriptor of any table, and the data changes we write should
> - * not be decoded by other backends.
> - */
> - LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
> - MyProc->statusFlags |= PROC_IN_CONCURRENT_REPACK;
> - ProcGlobal->statusFlags[MyProc->pgxactoff] = MyProc->statusFlags;
> - LWLockRelease(ProcArrayLock);
> -
> /*
> * The worker needs to be member of the locking group we're the leader
> * of. We ought to become the leader before the worker starts. The
i think as i did earlier in the diff, shouldn't we remove the duplicate code
from rebuild_relation, am i missing something?
--
Thanks,
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 156+ messages in thread
end of thread, other threads:[~2026-05-29 22:25 UTC | newest]
Thread overview: 156+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-04-01 14:55 Re: Adding REPACK [concurrently] Srinath Reddy Sadipiralla <[email protected]>
2026-04-01 17:06 ` Antonin Houska <[email protected]>
2026-04-01 21:52 ` Alvaro Herrera <[email protected]>
2026-04-03 12:08 ` Antonin Houska <[email protected]>
2026-04-03 14:59 ` Alvaro Herrera <[email protected]>
2026-04-03 19:31 ` Alvaro Herrera <[email protected]>
2026-04-03 19:37 ` Alvaro Herrera <[email protected]>
2026-04-04 06:20 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-04 10:19 ` 'Alvaro Herrera' <[email protected]>
2026-04-06 05:24 ` Amit Kapila <[email protected]>
2026-04-07 21:38 ` Alvaro Herrera <[email protected]>
2026-04-10 07:00 ` Alexander Lakhin <[email protected]>
2026-04-10 10:57 ` Antonin Houska <[email protected]>
2026-04-12 14:00 ` Alexander Lakhin <[email protected]>
2026-04-13 09:42 ` Antonin Houska <[email protected]>
2026-04-10 10:53 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-04-10 13:21 ` Antonin Houska <[email protected]>
2026-05-25 06:26 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-26 15:31 ` Alvaro Herrera <[email protected]>
2026-05-27 08:08 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-28 00:31 ` Amit Kapila <[email protected]>
2026-05-28 03:34 ` Amit Kapila <[email protected]>
2026-05-28 05:18 ` Zhijie Hou (Fujitsu) <[email protected]>
2026-05-29 15:39 ` Amit Kapila <[email protected]>
2026-05-29 22:25 ` Alvaro Herrera <[email protected]>
2026-04-21 07:24 ` Chao Li <[email protected]>
2026-04-04 13:16 ` Srinath Reddy Sadipiralla <[email protected]>
2026-04-04 13:55 ` Alvaro Herrera <[email protected]>
2026-04-03 17:24 ` Alvaro Herrera <[email protected]>
2026-04-04 08:50 ` Antonin Houska <[email protected]>
2026-04-04 09:48 ` Alvaro Herrera <[email protected]>
2026-04-04 15:29 ` Antonin Houska <[email protected]>
2026-04-04 23:53 ` Alvaro Herrera <[email protected]>
2026-04-05 07:54 ` Antonin Houska <[email protected]>
2026-04-05 11:50 ` Alvaro Herrera <[email protected]>
2026-04-05 16:30 ` Antonin Houska <[email protected]>
2026-04-05 18:56 ` Alvaro Herrera <[email protected]>
2026-04-05 20:41 ` Alvaro Herrera <[email protected]>
2026-04-06 09:15 ` vignesh C <[email protected]>
2026-04-06 09:38 ` Alvaro Herrera <[email protected]>
2026-04-06 10:01 ` vignesh C <[email protected]>
2026-04-06 11:23 ` Antonin Houska <[email protected]>
2026-04-06 11:56 ` Amit Kapila <[email protected]>
2026-04-06 21:11 ` Andres Freund <[email protected]>
2026-04-06 21:59 ` Alvaro Herrera <[email protected]>
2026-04-07 09:39 ` Antonin Houska <[email protected]>
2026-04-07 01:10 ` Noah Misch <[email protected]>
2026-04-07 01:58 ` Andres Freund <[email protected]>
2026-04-07 02:42 ` Noah Misch <[email protected]>
2026-04-07 04:44 ` Tom Lane <[email protected]>
2026-04-07 08:40 ` Alvaro Herrera <[email protected]>
2026-04-07 08:42 ` Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 08:57 ` Antonin Houska <[email protected]>
2026-04-07 09:22 ` Alvaro Herrera <[email protected]>
2026-04-07 09:35 ` Antonin Houska <[email protected]>
2026-04-07 10:08 ` John Naylor <[email protected]>
2026-04-07 10:30 ` Alvaro Herrera <[email protected]>
2026-04-07 09:29 ` Antonin Houska <[email protected]>
2026-04-07 09:41 ` Srinath Reddy Sadipiralla <[email protected]>
2026-04-07 11:19 ` Alvaro Herrera <[email protected]>
2026-04-07 12:17 ` Antonin Houska <[email protected]>
2026-04-08 10:20 ` Tomas Vondra <[email protected]>
2026-04-08 10:46 ` Antonin Houska <[email protected]>
2026-04-08 16:47 ` Tom Lane <[email protected]>
2026-04-09 06:59 ` Antonin Houska <[email protected]>
2026-04-06 10:14 ` Mihail Nikalayeu <[email protected]>
2026-04-06 10:48 ` Alvaro Herrera <[email protected]>
2026-04-06 11:21 ` Mihail Nikalayeu <[email protected]>
2026-04-06 22:22 ` Alvaro Herrera <[email protected]>
2026-04-06 23:53 ` Mihail Nikalayeu <[email protected]>
2026-04-07 12:15 ` Amit Kapila <[email protected]>
2026-04-07 12:33 ` Alvaro Herrera <[email protected]>
2026-04-07 14:10 ` Andres Freund <[email protected]>
2026-04-07 15:38 ` Antonin Houska <[email protected]>
2026-04-07 15:48 ` Andres Freund <[email protected]>
2026-04-07 16:01 ` Antonin Houska <[email protected]>
2026-04-27 04:25 ` Amit Kapila <[email protected]>
2026-04-27 04:46 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-30 11:24 ` Mihail Nikalayeu <[email protected]>
2026-05-04 13:24 ` Antonin Houska <[email protected]>
2026-05-05 12:47 ` Antonin Houska <[email protected]>
2026-05-05 15:02 ` Alvaro Herrera <[email protected]>
2026-05-06 08:25 ` Antonin Houska <[email protected]>
2026-05-08 12:25 ` Amit Kapila <[email protected]>
2026-05-08 13:58 ` Alvaro Herrera <[email protected]>
2026-05-08 23:22 ` Masahiko Sawada <[email protected]>
2026-05-10 11:24 ` Amit Kapila <[email protected]>
2026-05-10 11:31 ` Amit Kapila <[email protected]>
2026-05-11 15:17 ` Antonin Houska <[email protected]>
2026-05-11 19:30 ` Antonin Houska <[email protected]>
2026-05-12 07:57 ` Amit Kapila <[email protected]>
2026-05-12 11:08 ` Antonin Houska <[email protected]>
2026-05-13 12:34 ` Amit Kapila <[email protected]>
2026-05-13 16:58 ` Alvaro Herrera <[email protected]>
2026-05-14 07:02 ` Amit Kapila <[email protected]>
2026-05-19 18:52 ` Alvaro Herrera <[email protected]>
2026-05-23 15:29 ` Amit Kapila <[email protected]>
2026-05-24 11:29 ` Alvaro Herrera <[email protected]>
2026-05-12 02:25 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-05-05 15:04 ` Alvaro Herrera <[email protected]>
2026-04-07 12:32 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-04-07 14:19 ` Antonin Houska <[email protected]>
2026-04-08 03:49 ` Amit Kapila <[email protected]>
2026-05-08 23:22 ` Masahiko Sawada <[email protected]>
2026-05-01 07:30 ` 'Alvaro Herrera' <[email protected]>
2026-04-07 19:43 ` Robert Treat <[email protected]>
2026-04-08 07:31 ` Antonin Houska <[email protected]>
2026-04-07 20:24 ` Andres Freund <[email protected]>
2026-04-08 08:35 ` Amit Kapila <[email protected]>
2026-04-08 17:22 ` Andres Freund <[email protected]>
2026-04-08 23:36 ` Mihail Nikalayeu <[email protected]>
2026-04-09 04:54 ` Amit Kapila <[email protected]>
2026-04-09 08:43 ` Antonin Houska <[email protected]>
2026-04-09 09:11 ` Mihail Nikalayeu <[email protected]>
2026-04-09 09:26 ` Antonin Houska <[email protected]>
2026-04-09 14:06 ` Mihail Nikalayeu <[email protected]>
2026-04-09 14:26 ` Andres Freund <[email protected]>
2026-04-09 14:34 ` Andres Freund <[email protected]>
2026-04-09 14:20 ` Andres Freund <[email protected]>
2026-04-10 16:14 ` Robert Treat <[email protected]>
2026-04-10 18:56 ` Antonin Houska <[email protected]>
2026-04-12 14:02 ` Andres Freund <[email protected]>
2026-04-12 13:31 ` Mihail Nikalayeu <[email protected]>
2026-04-12 14:05 ` Andres Freund <[email protected]>
2026-04-12 14:58 ` Mihail Nikalayeu <[email protected]>
2026-04-14 13:37 ` Antonin Houska <[email protected]>
2026-04-14 13:58 ` Andres Freund <[email protected]>
2026-04-14 16:55 ` Mihail Nikalayeu <[email protected]>
2026-04-15 14:50 ` Antonin Houska <[email protected]>
2026-04-15 14:59 ` Andres Freund <[email protected]>
2026-04-15 18:28 ` Antonin Houska <[email protected]>
2026-04-16 11:18 ` Mihail Nikalayeu <[email protected]>
2026-04-17 02:01 ` Mihail Nikalayeu <[email protected]>
2026-04-18 19:23 ` Antonin Houska <[email protected]>
2026-04-18 22:46 ` Mihail Nikalayeu <[email protected]>
2026-04-20 17:44 ` Antonin Houska <[email protected]>
2026-04-23 11:43 ` Mihail Nikalayeu <[email protected]>
2026-04-26 13:34 ` Mihail Nikalayeu <[email protected]>
2026-04-19 11:00 ` Alexander Lakhin <[email protected]>
2026-04-17 14:44 ` Justin Pryzby <[email protected]>
2026-04-20 07:44 ` Antonin Houska <[email protected]>
2026-04-20 07:47 ` Antonin Houska <[email protected]>
2026-04-20 07:54 ` Chao Li <[email protected]>
2026-04-20 11:45 ` Alvaro Herrera <[email protected]>
2026-04-20 13:46 ` Antonin Houska <[email protected]>
2026-04-20 13:54 ` Alvaro Herrera <[email protected]>
2026-04-20 14:42 ` Tom Lane <[email protected]>
2026-04-20 15:11 ` Alvaro Herrera <[email protected]>
2026-04-20 16:01 ` Tom Lane <[email protected]>
2026-04-20 16:30 ` Alvaro Herrera <[email protected]>
2026-04-22 09:30 ` Alvaro Herrera <[email protected]>
2026-04-23 11:01 ` Alvaro Herrera <[email protected]>
2026-04-15 15:54 ` Mihail Nikalayeu <[email protected]>
2026-04-09 14:13 ` Andres Freund <[email protected]>
2026-04-06 09:03 ` Antonin Houska <[email protected]>
2026-04-02 00:19 ` Srinath Reddy Sadipiralla <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox